TSM Database and Recovery Log

TSM Database version 6.1 and later

Basic database structure

TSM 6.1 uses a DB2 relational database. This means it can be much bigger that the legacy database, does not neeed database audits and will automatically re-organise itself if required. Software database mirroring is no longer supported.
One thing to note is that you cannot have any other DB2 applications running on the same server as the one hosting your TSM database, though you can have multiple TSM databases on one server.

Several of the commands we used to use to manage the database are different. Some of these are :-

Database sizing and tuning

The database does not consist of a collection of files or 'volumes' like the legacy database. Instead, the database can exist in up to 128 directories or 'containers' to use the correct DB2 term. The data is striped evenly across the directories and unline the legacy DB, they do not require an initial format before they can be used. The Q DBSPACE output below shows a database striped over 3 containers.

 LOCATION               TOTAL SIZE OF File System(MB)  SPACE USED ON FILE SYSTEM (MB)  FREE SPACE AVAILABLE (MB)
 /tsm/tivoli/dbdir001   102,144                        6,919.22                        98,224
 /tsm/tivoli/dbdir002   102,144                        6,919.22                        98,224
 /tsm/tivoli/dbdir003   102,144                        6,919.22                        98,224

The database can be sized anywhere between 2.2GB and 1TB. The DB2 database will be between 35 and 50% bigger than the equivalent legacy database, partly becuase it hold sort space for SQL queries. The DB2 database is largely self tuning, so there is no requirement for DB2 tuning skills. A new parameter, DBMEMPERCENT, replaces the old BUFFPOOLSIZE. This set of buffers contains much more data than the old buffer so the recommendation is to set its size to unlimited. In fact, TSM/DB2 will try to change it to unlimited on startup.

Two other legacy features are not required now; database audits and indexed tables.
The database uses DB2 relational consistency rules to prevent incorrect data from entering, and is self auditing. The database will aslo run automatic 'runstats' from time to time. This is a DB2 feature that optimises storage paths through the database to improve performance.
The database also uses relational indices, so it does not require special index tables to speed up SQL queries.

Recovery log sizing and tuning

TSM 6.1 has two recovery logs.
The Active recovery log contains updates that have not been committed to disk yet and is used for roll-forward or roll-back in case of problems. Once a transaction is committed, the data is moved to the archive log. The default size for the Active log is 2GB and the size can be inceased by increments of 512MB right up to 128GB.
The Archive log contains committed transaction data and is used for PIT recovery of the database. The Archive log is cleared out by a full database backup. However it retains all data updates applied right back to the second last backup, so you need to size your archive log with that in mind.

The log files form part of the TSM database, and unlike the legacy TSM database there is no need to create and format log volumes. The logmode is equivalent to legacy roll-forward. In DB2 terms, these are archive logs, not circular logs. This means that the log files can fill up, so log file management is still required. You can specify a failover log for the Archive log to help prevent this, but the Active log cannot failover and the size is fixed between 2GB and 128GB, so don't allocate all the space that you have available for the Active log, keep some in reserve for emergencies.

If the Active log fills up and the server stops, the process to get your TSM server up again is:

DSMSERV DISPLAY LOG - to check the current log status
Update the Active log size parameter in dsmserv.opt
Start the server up

TSM Database version 5.5 and earlier

Recovery log processing
Database Defragmentation
Extending the TSM database under AIX
Formatting the TSM database and log
Auditing the TSM database
Database size and disk setup
Database and log Mirroring
Disk Storage Pools

Recovery log processing

The TSM database is quite sophisticated, and uses a transaction log, called the recovery log. Multiple updates are grouped together into 'transactions', which have to be applied to the database as a unit. If the server crashes, say, and the updates in a transaction have not been applied, then the partial updates must be backed out using the log records. This all or nothing approach protects database integrity during updates.

If the server cannot update the recovery log, because it is full, then the server crashes. So its worth knowing what makes the log fill up, and how to avoid it.

The log has two pointers, a 'head' pointer and a 'tail'. The head marks the position where the next update can take place, new updates are added at the head. The tail marks the position where the oldest transaction is still processing, and also where the last update can take place. Tail movement depends on how the 'logmode' is set up. If you define logmode=rollforward, then the tail will only move when a database backup is run. If you use logmode=normal, then the tail moves when the oldest transaction completes. When the pointers reach the end of the file, they start again from the beginning. Consider the logfile as being a circle, with the head and tail pointers being points on the circumference. The command Q STATUS will tell you which logmode you are using.

The tail is then 'pinned' by the oldest in-flight transaction, and if this is not cleared before the head catches up, then the file is full. Tivoli provided a new command with TSM 4.2.2.1, 'show logpinned', which will identify the transaction which is pinning the log.

The log file usually fills up due to a combination of two events. An old transaction hangs around and 'pins' the tail, while another process is causing the head to move rapidly, so it catches up.
Long running transactions can be caused by very large database backups, or smaller backups running over slow lines. A process which is trying to recover from a tape I/O error can also hang around for a long time.
Rapid head movement is caused by something which is doing large quantities of database updates, very fast. Expire Inventory is a good example of this. There are ways to manage this

• Don't schedule inventory expiration when large backups are running
• Make the log almost as large as possible, which is about 13GB at the moment. But, leave a bit of free space so you can extend the log if the server crashes.
• Consider clearing out your log before the backups start, by temporarily reducing the dbbackup trigger. UPDATE DBB LOGF=20 should force a backup. However, remember that if you are running with logmode=rollforward, and the tail is pinned, then the database backup will not clear out the log.
• Consider running with a smaller value of dbbackuptrigger during the backup run, to help prevent the log from filling. However, this can cause lots of backups to be triggered, so use with caution.
• Monitor the log utilisation, and alert support staff, if the log exceeds, say 80%. The support staff then need to look for an process which is holding the tail, and cancel it, or look for a process which is rapidly filling up the log and cancel that. Or, to be on the safe side, cancel them both.
• TXNGroupmax (maximum number of files sent to the server in a single transaction) and TXNBytelimit (total number of bytes in a single transaction) are usually set high to speed up backup performance. If you are getting problems with your log filling up, consider reducing these to force more frequent commit points.

Recovery log processing was enhanced in TSM 4.2. If the DB Backup Trigger is set correctly, and the LOGMODE is in ROLLFORWARD, then a database backup will start when the log reached 100% full. If the Recovery log hits 100%, then TSM will stop all processes except the database backup. When the backup completes, TSM issues the message

ANR2104I Activity log process restarted - recovered from an insufficient space condition in the Log or Database.

This should help us avoid some difficult recovery situations.

back to top

Database Defragmentation

This contentious issue applied to legacy databases only. The legacy TSM Server database has a b-tree format, and grows sequentially from the beginning to end. When file entries are deleted by expiration processing or file space/volume delete processing, this leaves spaces or holes in that database. These may be re-used later when new information is added, but they mean that the TSM database is using more space than it needs to. The only way you can compress the database so that the 'holes' left by deleted pages are not present is to use the database unload/reload utility.

The problem is that while the dump takes about an hour, the load utility can take several hours. Does it make a difference? I have seen performance improve after defragmenting a database, and I've also see an unload/reload make performance worse. A defrag will reduce the physical size of your database.

The Tivoli team supplied a new command with TSM 5.3 for to you to check to see what an unload/reload would achieve, called 'ESTIMATE DBREORGSTATS' This will estimate the amount of space that would be recovered by an unload reload.

For older releases of TSM use the QUERY DB to see if you need to defrag your TSM DB.

Available Assigned   Maximum   Maximum    Page      Total      Used   Pct  Max.
    Space Capacity Extension Reduction    Size     Usable     Pages  Util   Pct
     (MB)     (MB)      (MB)      (MB) (bytes)      Pages                  Util
--------- -------- --------- --------- ------- ---------- --------- ----- -----
   50,208   49,004     1,204     9,412   4,096 12,545,024 9,544,454  76.1  76.1
                                                                              

Here a 49GB database can be reduced by 9.4GB = 19%, but it is only 76% used, so 5% could be reclaimed by defragging. Some people claim that TSM tries to allocate pages in a way that leaves you with as good as possible performance, and defragging the database will degrade performance. Its also possible that after a defrag, the database will quickly become defragmented again, as it inserts data into the tree. The following formula can be used to see how much space could be reclaimed by an unload/reload.

SELECT CAST((100 - (CAST(MAX_REDUCTION_MB AS FLOAT) * 256 ) /
(CAST(USABLE_PAGES AS FLOAT) - CAST(USED_PAGES AS FLOAT) ) * 100) AS
DECIMAL(4,2)) AS PERCENT_FRAG FROM DB

A high PERCENT_FRAG value can indicate problems. If you think your database needs a defrag, then if possible, take a copy and try that first. That will give you an indication of how much time is needed for the load.

back to top

Extending the TSM database under AIX

Create a new file system in AIX using SMITTY

 	make LV
 	make FS on existing LV
 	mount new-filesystem

THEN in TSM

 
 	dsmadmc ... define dbv /new-filesystem/filename
 	dsmadmc ... extend db

If you use incremental database backups, then remember that after an EXTEND DB the next DB backup must be a full backup.

back to top

Formatting the TSM database and log

Legacy TSM database files and log files have to be formatted before they can be used. There are two different commands for this, and it is vitally important that you know the difference. If you want to add a file to the database or recovery log, then you use the DSMFMT command to format the file. The DSMSERV FORMAT looks similar but that command will format the whole recovery log and database. So just make things clear, DSMSERV FORMAT will wipe all your existing database and log files, so if you want to make a complete fresh start, that's what you use. DSMFMT will just format the file that you specify. The syntax of DSMFMT is

 dsmfmt -m -log tsmlogvol7 5

Which will format a 5 meg.log volume called tsmlogvol7. Size options are 'k' 'm' 'or 'g' and data type options are 'db' 'log' or 'data'

back to top

Auditing the TSM database

The Audit process only applies to legacy TSM databases.

Richard Sims has correctly pointed out that a database audit with FIX=YES is a dangerous procedure. "Correcting database problems without TSM Support direction can result in worse problems, including data loss. Structural problems and inconsistencies in any database system can be much more complex than a vanilla utility can properly deal with. If one has a reason to believe that their TSM database has problems, they need to contact TSM Support for assistance in dealing with them, rather than attempt amateur surgery. IBM repeatedly advises customers NOT to attempt to fix database problems themselves".
I'd also suggest that if you run an audit, you always make sure you have a full database backup available first.

Database Audits are used to fix inconsistency problems between the database and its storage components. A full database audit can run for several hours, but it is possible to run smaller audits on parts of the database. As a general rule of thumb, a full database audit takes about 3 hours per million pages, and a 4 GB utilised database holds about a million pages. The actual times will mostly depend on the processing power of your server. An audit will write a lot of log records so if you normally run with your recovery log in 'ROLL FORWARD' mode it is advisable to put the log into 'NORMAL' mode before running an audit, then put it back into 'NORMAL' mode when the audit completes.

/dsmserv auditdb fix=yes admin detail=yes 

Is a very quick check of the admin data

/dsmserv auditdb fix=yes archstorage detail=yes

will audit the archive storage, and runs for 1-2 hours depending on your database size

/dsmserv auditdb fix=yes diskstorage detail=yes

will audit the disk storage pools, and takes about 30 mins, depending on the size of the database, and how full the disk pools are. Best done when all the data is migrated out to tape.

/dsmserv auditdb fix=yes inventory detail=yes

This is the long running one, 8-12 hours.

The following information was supplied by Maureen O'Connor of Fiserv Technology in April 2007. Maureen has provided some excellent detail on how to estimate how long an aufit will take, and how to run audits against multiple TSM servers on one AIX server.

ANR4134I AUDITDB: Processed 8260728 entries in database tables and 0 blocks in bit vectors.  Elapsed time is 1:05:00.
ANR4134I AUDITDB: Processed 9035641 entries in database tables and 0 blocks in bit vectors.  Elapsed time is 1:10:00.
ANR4134I AUDITDB: Processed 9812999 entries in database tables and 0 blocks in bit vectors.  Elapsed time is 1:15:00.
ANR4134I AUDITDB: Processed 10663992 entries in database tables and 0 blocks in bit vectors.  Elapsed time is 1:20:00.
ANR4134I AUDITDB: Processed 11677212 entries in database tables and 0 blocks in bit vectors.  Elapsed time is 1:25:00.
ANR4134I AUDITDB: Processed 12014759 entries in database tables and 0 blocks in bit vectors.  Elapsed time is 1:30:00.

select sum(num_files)*3 from occupancy

AIX# export DSMSERV_DIR=/usr/tivoli/tsm/server/bin
AIX# export DSMSERV_CONFIG=/usr/tivoli/tsm/server/bin/<subdir>/dsmserv.opt

AIX# at now
dsmserv auditdb fix=yes admin detail=yes > /tmp/tsmauditadmin.log
[ctl-D]

dsmserv auditdb fix=yes archstorage detail=yes >/tmp/tsmauditarchive.log

dsmserv auditdb fix=yes diskstorage detail=yes >  /tmp/tsmauditdisk.log

dsmserv auditdb fix=yes inventory detail=yes >  /tmp/tsmauditdata.log

ANR1777I afaudit.c(967: Object 0.62882489 is \WINDOWS\INF\DSUP.PNF for node
<NODENAME> (257), filespace \\<nodename>\c$ (1).
ANR1777I afaudit.c(967: Object 0.62882490 is \WINDOWS\INF\DSUPT.PNF for node
<NODENAME> (257), filespace \\<nodename>\c$ (1).
ANR1777I afaudit.c(967: Object 0.62882491 is \WINDOWS\INF\DVD.PNF for node
<NODENAME> (257), filespace \\<nodename>\c$ (1).
ANR4303E AUDITDB: Inventory references for object(0.62882489) deleted.
ANR4303E AUDITDB: Inventory references for object(0.62882490) deleted.
ANR4303E AUDITDB: Inventory references for object(0.62882491) deleted.

back to top

Legacy Database size and disk setup

The TSM database is critical to open systems backup and recovery. It needs to be 100% available as without it, it is impossible to recover files. The 'incremental forever' philosophy behind TSM means that it is impossible to build a list of files needed to recover a server without the TSM database. If the TSM database setup is not designed correctly then the database will perform badly and this will affect your ability to fit backups within the overnight window.

TSM performance is very much dependent on the size of the database. TSM performance suffers if a database becomes too large, but there are no exact rules on how big too large is. The maximum possible size for a TSM database is 530GB. IBM recommend 120 GB as a general rule, with the caveat that 'when expiration, database restores, and other Tivoli Storage Manager admin processes take too long and client restores become too slow, it is too big'. Database backup and Expire Inventory are both CPU intensive processes that can be used to indicate server performance problems in general. The only sensible answer to 'how big should a TSM database be?' is to let you database grow until these processes start to become an issue. Expire Inventory should really run within 12 hours and should be processing 3 million pages an hour or more. Backups should run in 30 minutes and process 6 million pages per hour or more, but these are just general rules-of-thumb. The actual size will depend on how fast your hosting server is, how good your disks are and what level of service you need to provide.

A TSM Database consists of a number of files, called 'disks'. As TSM will schedule one concurrent operation for each database disk it makes sense to allocate a lot of small disks, rather than a few large ones. A disk file size of 2 GB seems to be about right (The maximum possible size for a disk volume is 8 TB). IBM recommends that these database disk files be spread over as many physical disks as possible. This makes sense for low or mid tier disk subsystems, as this means that multiple disk heads can be seeking, reading, and writing simultaneously, but as high tier subsystems perform most of their I/O in cache this is less of an issue.

Most operating systems allow you to stripe files over logical and physical disks, or partitions, and recommend that this be used for large performance critical files. It is very difficult to get any kind of consensus from the TSM user community on the benefits of disk striping. For example to quote two users:-
USERA; 250GB! database on a high tier EMC DMX disk subsystem. Disk striping introduced and database backup reduced by more than half.
USERB; 80GB database striped on a mid-tier IBM FASTT subsystem striping removed and database converted to RAID5. No impact on database backup times, expire inventory run times or client backup times.

TSM will allocate a default database and log file during a AIX usually in the server installation directory /usr/tivoli/tsm/server/bin These default files should be deleted and re-allocated to your strategic size and location.

back to top

Database and log Mirroring

There are three levels of mirroring, Hardware controlled, Operating Systems controlled and TSM controlled.

Mirroring protects the database from disk failure, and also subsystem or site failure if the mirroring is between subsystems or sites. Mirroring also offers some protection from system failure as the chance that at least one of the mirror writes was successful is much higher. TSM mirroring can detect if a partial write has occurred then a mirror volume can be used to construct valid images of the missing pages. TSM mirroring can complement hardware mirroring. It is best to mirror both the database and the recovery log to optimise availability and recoverability.

If you are using automatic database or logfile expansion with mirroring, then this will place both the primary file and the mirrored file in the same directory, as only one directory path can be specified. This means that the primary file and mirrored file could end up on the same disk, so they will need to be separated.

This sounds obvious, but the mirrors need to be on different disks. It is possible to place them on the same disk and that would be pretty pointless. It is also possible to mirror to three ways as well as two ways. With three-way mirroring you get three copies of the data.

Hardware mirroring (RAID1)

Most disk subsystems support RAID1 mirroring, which is expensive as it needs twice as much disk, and will not detect logical errors in the data. All data is mirrored even if it is corrupt.

Operating System Mirroring

IBM state that disk striping is suitable for a large sequential-access files that need high performance. AIX supports RAID0, RAID1 and RAID10. RAID0 is not really a good idea, as if a logical volume was spread over five physical volumes, then the logical volume is five times more likely to be affected by a disk crash. If one disk crashes, the whole file is lost. RAID1 is straight disk mirroring with two stripes and requires twice as much disk. RAID10 combines striping and mirroring, and also uses twice as much disk.
If AIX is mirroring raw logical volumes it is possible for it to overwrite some TSM control information, as they both write to the same user area on a disk. The impact would be that TSM would be unable to vary volumes online.

TSM mirroring

Software mirroring just applies to the legacy database. If TSM is managing the mirror and it detects corrupt data during a write, it will not write the corrupt data to the second copy. TSM can then use the good copy to fix the corrupt mirror. TSM also mirrors at transaction level, and hardware at IO level. Hardware will always mirror every IO, but TSM will only mirror complete transactions. This also protects the mirror from corruption.

back to top

Disk Storage Pools

TSM will only perform one concurrent IO operation to every storage pool volume, so it is better to have a number of smaller volumes than a single large volume in your disk storage pools. Also, it is easier to relocate a small volume to a different disk pool if space requirements change. However, every volume will use one TSM processing thread, and the TSM server will crash if too many threads are allocated.

The normal process is to initially write backup data to disk, then move it off to tape. It is possible to copy the data to tape but not delete it from the disk pool, if the disk cache setting is set to 'yes'. The TSM server would then use the disk data for restores and delete it as space is needed for fresh backups. This would speed up recovery, but slow down backups as the TSM server has to do extra work clearing out data, and would also make the TSM database bigger as TSM needs to store two locations for recently backed up data. It is your choice, faster restores or slower backups and a bigger database.

back to top

Copyright © Lascon Storage Ltd. 2000 to present date. By entering and using this site, you accept the conditions and limitations of use