IAM, a VSAM Alternative
IAM, an alternative to VSAM
TSM Restore Hints
Advantages of IAM over VSAM
IAM (Innovation Access Method), is non-VSAM, but looks like VSAM to application programs. So why buy IAM, when VSAM comes with z/OS? The basic advantages of IAM are :-
- VSAM needs multiple buffers to perform. IAM does this using a concept called IAM Real Time Tuning, which keeps the file index in virtual storage. This means that all of the index I/Os and index buffers used by VSAM are eliminated. IAM requires, at most, one I/O to retrieve any record identified by key within a data set. This reduces the need to perform physical I/O to look up and retrieve data. Comparison between IAM and VSAM on real production data, can see batch run times were reduced by 40%, even if the VSAM files are buffer tuned.
A cautionary note. A batch job will typically only open one or two files at a time, so storage requirements are low. An on-line CICS system can have hundreds of files open at the same time, and this can required lots of virtual storage.
- IAM can speed up error recovery. It can automatically journal file updates for Enhanced Format IAM KSDS and ESDS files. This is especially useful for batch failures, where recovery would mean restoring to the last good backup, then re-running batch work to the point of failure. This can take hours. The IAM journal allows you to backout a failing job by rolling back the log updates to get back to the start of the job. In comparison tests, recovery times reduced from an average 2.5 hours to an average 30 minutes
- IAM eliminates the 4GB limit for normal VSAM files. The actual file size available will depend on the DASD type, and DFSMS release, but its possible to get a 200GB IAM file with compression on IBM 3990 format DASD. Its been possible to extend all VSAM file types since OS/390 V2.10, but these all have to be SMS managed. IAM does not.
- VSAM files tend to waste space, due to the requirement that block sizes are a multiple of 512 or 2048. It is also difficult to release unused space from a VSAM file. With IAM, the blocksize is still determined by the CIsize if you allocate the file with IDCAMS, but you can use half track blocking. Also IAM's different file structure and use of compression means that data stored in an IAM file typically requires substantially less DASD space than when stored in a VSAM cluster. InnovationDP claim that IAM reduces space utilisation by 20 - 40%. IAM uses a propriety compression engine that gets the best trade-off between disk space reduction and CPU usage. IAM 8.1 included an enhancement to use z/Series hardware compression instead of IAM's native compression. This may be less space efficient, but it reduces CPU requirements.
- Key sequence datasets are stored in key order by definition, and VSAM requires that free space be maintained in each Control Interval to insert records. IAM honours this free space concept by reserving free space in an allocation block for new records. However VSAM also reserves free space in a Control Area and once this free space is used up, it is necessary to split the Control Area, moving half to the end of the file. If IAM cannot add a new record to an existing block, it will add the record to an overflow area. The overflow area is indexed separately, and can reuse space freed up by deleted records. The overflow index is held in virtual storage for fast access. There are two types of IAM file, Enhanced Format and Compatible Format. With Compatible format the overflow area is defined up front and cannot be extended. With Enhanced format, extra disk extents will be dynamically added to the overflow area as required. The IAMINFO report will tell you how many overflow records are in use, and how many are free
VSAM suffers performance degradation when it needs to split CIs and CAs. A consequence of the way IAM handles record inserts is that unlike VSAM, it does not suffer from CA splits and CI splits.
IAM installation and file structure
IAM interacts with programs that would normally use VSAM, like Cobol for instance, using the VSAM Interface (VIF). The VIF is dynamically allocated in the SVC table at IPL time, and then it intercepts SVC calls like VSAM OPEN and VSAM CLOSE. If it finds that the request is for a VSAM file it just passes control to the normal IBM routines and they process the VSAM calls as normal.
If the request is for an IAM file, it opens the IAM ACB in a way that makes it look like a VSAM file, then, when the application program issues a VSAM record request (PUT, GET etc.), this request is converted by the VIF into commands to process the IAM file. This is completely transparent to the application, as far as it is concerned it is using standard VSAM commands to a file that looks just like VSAM.
An IAM file consists of the following components:
- The front of the file contains the 'self-describing' control information such as Blocksize, Record format and logical record length.
- Next comes the Prime Data Area. This contains the data blocks that are initially loaded into the file and these blocks may contain Integrated Overflow freespace to allow records to be inserted. This is NOT like a CI split, no data is moved to create freespace.
- The Index comes next, after the Prime Area. The index is loaded into storage when the file is opened to enhance performance.
- The end of the file contains the Extended Overflow and Extended PE blocks. Their usage is explained below. IAM can use Secondary extents if it needs more disk space for overflow blocks.
A VSAM KSDS is split into Control Areas (CA) and Control Intervals (CI) and free space is maintained in both of these to avoid CA and CI splits. This is explained in detail in the VSAM structures page
An IAM file is allocated in blocks that are determined by the CIsize parameter in the IDCAMS define command, but these blocksizes are usually much bigger than the ones used by VSAM. While IAM does not have control interval and control area concepts, it uses the FREESPACE(CI% CA%) as follows.
CI% is used to calculate the amount of space to keep free in each block for an Integrated Overflow area when the file is initially loaded. As new records are inserted into a file they are placed into the appropriate Integrated Overflow area, and when that area fills up, they are placed into an Extended Overflow block at the end of the file
CA% is used to decide how much DASD space to release at the end of a file load. Inserts that go to the end of an IAM file because their keys are higher than existing records are allocated into Extended PE blocks.
Prime Related Overflow was introduced with IAM 8.1. and is an option for IAM files that are exposed to a very high volume of records being inserted, especially if those inserts are clustered around a certain key range. This matches overflow blocks with primary data blocks and so reduces the requirement for overflow indexing, which in turn improves performance.
The in-Core Index is possibly the biggest IAM performance enhancement. When an IAM file is opened, IAM places the entire INDEX of the file into Virtual Storage, so every subsequent access to that index comes from storage and does not require any IO.
IAM uses dynamic buffering to reduce the real IO requirement for the Data area. To achieve this, IAM monitors the IO access pattern to a file and then adjusts the number of data buffers as required. IAM uses various techniques to achieve this, such as random or sequential processing detection and LRU algorithms.
IAM also has an optional 'TURBO' mode than can acquire buffers much more aggressively if it detects heavy IO activity.
IAM can also schedule multiple-concurrent IOs to multi-volume files. This technique is used to speed up processing by FDRREORG when reorganising multi-volume IAM files.
Another IAM feature is Dynamic Tabling. Here, IAM creates a 'table' in virtual storage that contains any records that are read from a file. If the record is requested again, IAM retrieves it from the table instead of having to go out to disk with a real IO. This will obviously not besuitable for every file but works well for files that have a high read, low update access pattern, especially if the reads are random and the same records are continually read.
IAM provides RLS functionality, and it also has a journalling facility to log batch updates that are made to IAM files. This can be used to back-out failed batch jobs, similar to VSAM TVS. The back-out can be automated, so if a job abends, its updates are automatically removed from the IAM file, either back to the beginning of the job, or back to the last sync point. The journalling does not affect CICS as CICS has its own journalling system.
IAM can write both BEFORE and/or AFTER journal records when an IAM file is updated. Typically, these are used for:
If a batch job fails while updating an IAM file, the data will probably be in an indeterminate state, or corrupt. The BEFORE images can be used to backout the updates in reverse order, so getting the file back to the state it was in before the failing job started.
A 'forward recovery' is used if an older copy of an IAM file has to be restored, perhaps after encountering a media failure, or data corruption, or maybe even in the event of Disaster Recovery. Once the older copy of the file is back on disk, the “After” images can then be re-applied to the file, working forwards in time, thus recreating the updates that had taken place.
These procedures, which have been available for some time in IAM, have been enhanced to accommodate the changes introduced by the IAM/RLS and IAM/PLEX.
This can be automated with IAM's 'Dynamic Job Backout' (DJB) facility, which provides a controlled, automated and fully dynamic “backout recovery” of IAM/RLS and IAM/PLEX files. If an updating job step abends, all uncommitted updates (for that job) are automatically removed from all IAM/RLS or IAM/PLEX files being updated by that job. This negates the need to run separate, manual, batch-driven recoveries for each affected file.
IAM and RLS- Record Level Sharing in a single LPAR
The principle behind Record Level Sharing (RLS) is explained in the VSAM RLS section. IAM implements RLS at two levels: The original IAM/RLS implemented record sharing within a single z/OS image or LPAR as they are usually called. IAM/PLEX was introduced in IAM Version 9.0 as an extra cost-option. This permits applications running in either the same LPAR, or different LPARs in a SYSPLEX to concurrently access an IAM file and is discussed in the next section.
VSAM RLS and TVS require that z/OS be part of a SYSPLEX, and store the sharing and log records in the coupling facility. IAM/RLS does not do this; it uses an IAM RLS address space and journalling datasets. IAM implements RLS by passing all I/O requests for the files it is sharing to an IAM RLS address space. It then uses a lock manager to handle locking requests.
IAM decides which files are eligible for IAM/RLS processing based on a combination of the file share options, an RLSID parameter that is set when the IAM file is defined, and a special IAM/RLS data set name table.
IAM/RLS requires that some exits be installed in CICS but little in the way of JCL changes are required. Concurrent update processing is managed by using locks at record level, utilizing the IAM/RLS record locking facility. For recoverable files, these record locks are held until the job or transaction terminates, or the job or transaction issues a SYNCPOINT. This means that batch jobs that do lots of updates should have Batch Syncpoint and appropriate restart capabilities installed.
An ISPF interface is provided to monitor activity within the IAM/RLS address space
IAM/PLEX - Record Level sharing in a SYSPLEX
IAM/PLEX is and extra cost option on IAM 9.0, and uses the Cross Coupling Facility (XCF) service to manage record level data sharing in IAM files between applications in a SYSPLEX. The applications can be a mixture of CICS transactions, batch programs, or TSO users.
IAM/PLEX is basically just an extension of IAM/RLS and so it is much easier to implement than native VSAM file sharing. CICS requires 2 IAM exits to be installed, but no changes are required to application programs. TSM and batch programs that have a heavy write activity need syncpoints adding, but otherwise, no changes are required.
An IAM/PLEX 'instance' runs as an address space on all the z/OS LPARS that need to share IAM files in a SYSPLEX. These instances combined to make IAM/PLEX RLS group. Each IAM dataset is owned by one of the IAM/PLEX instances and all I/O to that IAM dataset goes through that owning instance. Record locking is managed by the owning instance and this ensures data consistency, even thought the file might be updated by applications running on several LPARs. The I/O requests from the other LPARs are routed to the owning instance through the SYSPLEX XCF.
IAM datasets use journaling for forward or backward recovery, and journaling for shared IAM file use the shared z/OS sysplex logger.
Alternate Index support
Alternate Index support for IAM can be purchased as an extra option and can be used on KSDS, ESDS and RRDS file types. If you need alternate indexes then you define them and their paths using the standard IDCAMS DEFINE ALTERNATEINDEX, then you build the index with the BLDINDEX command.
IAM hints and tipsIAM does not work well with DFSMS compression, so the SMS dataclass should be defined with COMPACTION (NO).
IAM should be totally transparent to applications, and system commands. However Innovation does recommend that you add an IAMINFO DD statement to every batch job to aid in any required problem determination. IAM supports KSDS and ESDS files. Older versions of IAM did not support KSDS files with alternate indices, but this is supported with IAM/AIX V7.0
IDCAMS commands are supported for IAM as they do for VSAM, so no maintenance JCL changes are needed. However, only DELETE, PRINT, REPRO and VERIFY work the same with both IAM and VSAM. LISTCAT ALL displays IAM files as non-VSAM in its standard SYSPRINT report. LISTCAT ALL also displays the file's IAM characteristics in an IAMPRINT DD report, which will be dynamically allocated if necessary.
IAM files are in proprietary format, therefore only systems which have IAM installed can successfully process the data in them. If you share data between several LPARS, then you need IAM licensed on all LPARs which will access the data. If you don't do this, then the data will appear to be corrupt from the unlicensed LPAR. This makes life 'interesting' when testing.
There are several was to create an IAM file
- Use an IDCAMS DEFINE either with parameter OWNER(£IAM) specified or with £IAM somewhere in the data set name.
- Use a standard DISP=NEW,CATLG JCL allocation with £IAM in the dataset name, or with an SMS Data Class or Storage Class that has £IAM in the name.
- Allocate an IAM file using the IAM ISPF panels
The £IAM is UK standard, non-UK implementations need to use $IAM. While any of these methods will work, the £IAM in the file name makes it immediately obvious that a file is IAM, and so is a good standard.
Sample IDCAMS JCL to define an IAM ESDS with £IAM in the name
//Your job card //STEP01 EXEC PGM=IDCAMS //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //SYSIN DD * DEFINE CLUSTER - (NAME(SAP1.£IAMCICS.A23RT) - MODEL(SAP1.PR01CICS.A13T) ) - DATA (NAME(SAP1.£IAMCICS.A23RT.DATA) - CYL(250,250))
OR to define a KSDS using the OWNER parameter
DEFINE CLUSTER - (NAME(SAP1.PR01 CICS.A23RT) - MODEL(SAP1.PR01CICS.A13T) - OWNER(£IAM) ) - DATA (NAME(SAP1.PR01CICS.A23RT.DATA) - CYL(250,250) - INDEX (NAME(SAP1.PR01CICS.A23RT.INDEX) - CYL(25,25))
The maximum size of an IAM file is about 3 TB if allocated in extended format on a 3390-54.
The maximum record size will depend on the key length and offset, but if the IA file is in Enhanced format and had the SPANNED attribute then the maximum record size is about 8 MB. Compatible mode files are restricted to 32,755 (variable length) or 32,760 (fixed length).
Although IAM files are physical sequential (DSORG=PS), indexed IAM files need to be reorganised the same as KSDS VSAM files. In the case of IAM, this is to reduce the amount of data that is stored in the overflow areas, as a large overflow area will require a large amount of virtual storage. Enhanced format files should be able to last longer than compatible format IAM files, as the overflow area can be extended. However, Innovation recommends that if you have 10% or more of your records in the overflow area, it should be reorganised. Like any other dataset, IAM files can fill up and cause applications to fail with an SB37 error.
IAM itself will issue an IAMW22 warning message if it thinks a reorganisation is needed. The reorg. process can be automated using FDRREORG, or you can use the normal IDCAMS REPRO process if you are not licensed for FDRREORG.