Windows Storage - Defragmentation

What is Fragmentation?

When a Windows operating system is first installed on a disk and then applications are added, all the files are contiguous and arranged in the optimal place on the disk for performance. This means that the read-write heads do not need to move much to read a single file. A file is fragmented when it is not contiguous, but is split up into pieces in various parts of the disk. Now the read-write heads have to move back and forward over the disk to read the whole file. This takes time, and causes 'seek delay' performance problems.

When a file is updated, it usually needs space to grow. The NTFS file system leaves space before and after a file, to allow it a bit of room to grow without causing fragmentation. Microsoft initially thought that this meant that the NTFS was fragmentation proof, and were so confident that they did not supply a defragmentation tool. However, this is not quite correct. The 'guard space' which NTFS leaves on either side of files is about 30% of the space used by the file, so once an NTFS partition hits the 70% full mark, the data has actually hit the end of the disk. When more new files are added, they will be allocated into the 'guard bands', and may well be fragmented.
Over time, as files are deleted and new files added, the free space becomes fragmented, so it is impossible to allocate a new file into a single space. Fragmentation and performance will get worse with time.

In the real world, performance does not die when partitions are loaded beyond 70%. This is probably due to a combination of the other NTFS performance features, and also that the big cache in RAID disks mask sequential access and seek problems (because the files are no longer allocated in nice contiguous chunks, but are split up and striped over several disks).

A defragmentation tool basically works by moving files around on the disk, so each file becomes consolidated into one contiguous extent, and to consolidate the freespace into a single contiguous extent for new allocations. This ideal cannot be achieved in practice, as some files cannot be moved.

Unmovable files

Files which cannot be moved with Windows active are
Bootsect.dos
Safeboot.fs
Safeboot.csv
Safeboot.rsv
Memory.dmp
FAT directories
Change journal
and any other files open for exclusive use.

Master File Table

The MFT is an index file that maps everything stored on a disk, and is usually held right in the centre of the disk. The MFT is explained in the NTFS page. As MTF entries are being constantly added or deleted as files are created or deleted, the MTF gradually becomes fragmented, resulting in longer disk access times and diminished performance.

Paging File

Windows uses the Paging File to swap data in and out of Random Access Memory (RAM). When the RAM fills up, Windows needs to create more room to load programs or data, so it swaps the oldest data from memory out to the paging file on disk. If the Paging File becomes fragmented, performance will suffer, and you can usually hear the hard drive thrashing while you wait for a response.

Hibernate File

The Hibernate File is where memory data is stored when the computer is turned off. Because the location of the Hibernate File is determined very early in the boot process, it cannot be moved.

It is impossible to defragment any of the three files above using normal Windows compression tools, you need a third party tool which works at boot time

Directories can also become fragmented; adding to the time it takes to access files. Windows 2000 directories on an NTFS disk can be defragmented while the operating system is running. However, FAT directories can only be defragmented at boot time.

Good defragmentation policies

• Delete any files that you don't need before starting a defrag
• Run a chkdsk to make sure your disk is not dirty.
• Make sure you have at least 15% free space to be used as a sorting area
• Kill off as many active processes as possible to minimise files in use
• Do regular defrags.

back to top

The Windows Disk Defragmenter

Microsoft provided a system defragmenter tool from Windows 2000 onwards. It is based on the Diskeeper product described below. It has two functions, an analyze function to determine if a disk needs defragging, and the defragmentation function itself. To run a defrag -

Click Start - Programs - Accessories - System Tools - Disk Defragmenter; Select the volume that you want to check for fragmentation; Click Analyze and the message box will tell you if a defragment is necessary. If so, click Defragment.

A few pointers -

• It is best not to do other tasks while Disk Defragmenter is defragmenting a volume, because response times will be poor, and because if you hold exclusive locks on files, they will not be consolidated.
• Defragment will not run if the disk has any errors. You may want to use Scandisk to check this first.
• While the defragmenter is running, you can see a detailed colour map of what's happening. Check out the Legend box to see what the the colours mean. Each coloured pixel can represent thousands of clusters, each of which can be one of several states. The pixel value represents the state of the biggest number of clusters in that group.
• You need a reasonable amount of free space to run a defrag successfully. Microsoft recommend 30% free space, excluding any free space reserved for the MFT.
• A defrag can take several hours. Defragmentation speed is related to the number of files on a disk, so it will take much longer for large, full disks.

Third party tools

Microsoft introduced a File System Control (FSCTL) command FSCTL_READ_MFT_RECORD for defragmentation in Windows NT 4.0.
Third party developers use FSCTL_READ_MFT_RECORD to determine what files are on a NTFS volume and where their clusters are located. They can combine this with other FSCTL commands like get_volume_bitmap, get_retrieval_pointers and move_file to develop defrag programs without knowing anything about FAT or NTFS data structures, or without even needing to know which type of file system is being processed

back to top

Diskeeper

Diskeeper is supplied by Executive Software, and in fact, the Microsoft defrag tool used Diskeeper code. So why would you buy a third party tool when Microsoft provide one with the operating system. Executive Software claim the following advantages.

• You can't normally defragment your directories because they're always in exclusive use. Diskeeper defrags your directories at boot time, which might mean the boot takes longer, but it can improve performance.
• Diskeeper can defragment the paging file and the MFT at boot time. Its also possible to schedule this for remote servers using remote boot time scheduling. Diskeeper V9 introduced FragShield, a utility that will dynamically prevent fragmentation of critical system files.
• Executive Software claim that Diskeeper is the fastest Windows defragmenter ever built, 300% to 500% faster than the Windows Disk Defragmenter. It can identify the areas of a disk that are causing performance problems and just defrag them.
• Diskeeper has extensive scheduling facilities, including 'reschedule on miss', and remote scheduling including centralised reporting of results of remote operation. The native Windows defragmentation tool is a manual operation. Diskeeper can schedule groups of disks of any size or all disk volumes on computers, workgroups or domains throughout the network. It also enables direct connection to a specific computer on the network and remote execution of all Diskeeper operations including analyzing and defragmenting disk volumes, setting schedules, configuring exclusion lists and setting run priorities. A Command Line Interface is also available. Diskeeper V9 introduced SmartScheduling which will check disk usage patterns and automatically schedule a background defragmentation if required. It is possible to combine SmartScheduling with ScreenSaver scheduling to get optimum results.
• Diskeeper has three defragmentation method algorithm options: Maximum Disk Performance (recommended), Quick Defragmentation and Improved Free Space. It is designed to run in the background unnoticed by end-users, depending on the priority setting. There are 5 priority levels, the default is to run as a low priority background task, which is almost undetectable with Diskeeper version 9 as it includes a High-speed defrag engine with I/O Smart technology.
• Diskeeper can defragment multiple drives or volumes simultaneously, whereas the Windows tool can only defragment one local drive volume on a machine at a time. The Terabyte Volume Engine, available with Diskeeper Server Enterprise Edition, is especially designed to defragment very large volumes.

Diskeeper version 9 has different editions for Home PCs, XP Professional, Servers or enterprise. Windows XP support.

The Home edition will defrag automatically in the background when the fragmentation level exceeds a threshold and will graphically display benefits and results.

The Professional edition does boot time defragmentation for critical system files and can defrag up to 4 drives simultaneously.

The Server edition will defrag up to 8 drives simultaneously. When combined with the Diskeeper Administrator it will schedule defrags and report on success

The Server Enterprise edition will defrag very large volumes, up to 4 terabytes, and can do 16 simultaneously. It can also can be managed with Diskeeper Administrator. This edition is required for Windows Server 2003 Enterprise Edition and Windows 2000 Advanced Server

back to top

PageDefrag

PageDefrag is a freeware utility that is specifically designed to let you defragmented your paging files and Registry hives, event log files and Windows hibernation files. This could make it an ideal complement to the Windows Defragmentation tool. It supports Windows systems from NT 4.0 upwards.

When you run PageDefrag, it gives you a listing of how many clusters make up your paging files, event log files, and Registry hives as well as how many fragments those files are in. You then select a radio button to decide if you want to defragmented the files at the next reboot. Immediately after CHKDSK examines your hard drives PageDefrag uses the standard file defragmentation APIs to defragment the files. As it processes each file PageDefrag will print on the boot-time startup screen the file name and its success at defragmenting it. If it is successful at reducing the fragmentation it will tell you the number of clusters the file started with and the number it consists of after the defragmentation.

In some cases PageDefrag may be unable to reduce fragmentation on one or more of the files, and it will indicate so on the boot-time Blue Screen. This can happen either because there is not enough space on the drive for defragmentation, or the free space itself is highly fragmented. For the best results you should use PageDefrag in conjunction with a general purpose defragmentation utility.

back to top

O&O Defrag V8

O&O Defrag V8 works with Windows NT 4 and above. It has the usual defrag facilities, it can defrag systems files at boot time, and it will run in 'screen saver' mode to automatically defragment disks when a computer is not is use. It integrates with Windows explorer to present a simple user interface. It will defragment an unlimited number of drives or partitions, and can defragment almost any type of disk including USB sticks, memory cards and RAID devices, with only about 5% free space needed for operation (the Windows defragmenter needs 15% free space).

O&O Defrag V8 offers you five different methods of defragmenting your computer:
STEALTH, SPACE, COMPLETE/Access, COMPLETE/Modified and COMPLETE/Name
All of these methods will work alongside working applications and services. The main differences are that Stealth and Space are optimised for speed, and the Complete variants are optimised for completeness.

STEALTH is designed for very large disks, greater than 1TB, and SPACE is designed for large disks, greater than 100 GB. SPACE needs more initial free disk space than STEALTH. Either method is suitable for the first defrag of a badly fragmented disk, or for where server CPU and memory resources are limited.

COMPLETE/Access will sort files into the order in which they were last accessed, COMPLETE/Modified will sort files into the date they were last changed, and COMPLETE/Name sorts the files alphabetically. Complete/Modified is suitable for database servers, and COMPLETE/Name is suitable for disks where speed of access to files is most important. All these defrag methods will require a lot of CPU, memory and disk resource.

You can schedule pre and post defrag Windows commands if required, and a 'OneButtonDefrag' option will run an automated defrag with a single mouse click. The product also has an 'ActivityGuard' component to ensure that it does not use up too much system resource.
The server version of O&O Defrag extends this functionality to the whole Windows Server network.

Once you install O&O Defrag agent, all communication is controlled with a network management console. Whenever the console recognizes a new computer on which the agent has not yet been installed, installation can be either automated or the administrator is automatically prompted to install it.
You can manage all machines consistently as a group, or you could apply individual settings to each machine. You can define schedules to automate defragmentation over all Networked clients, and you can schedule pre defrag and post defrag Windows commands. The Server version also has a 'OneButtonDefrag' function for the automatic defragmentation of entire groups of client computers on a network.
You can also give your users some control over the defragmentation process on their own workstation.
You can display an icon on the client's desktop that lets the users know about the current defragmentation job and allows them to pause or even stop it.
You can also give your users access to the full O&O Defrag GUI, so they can change the configuration parameters for their own workstation, without affecting the global parameters on the Network.

Generally, busy files should be near each other to minimise head movement, and files near the centre of the disk get the best performance. O&O Defrag V8 allows you to rearrange files in a specific order, so you can get the files you think are important in the optimal position.

O&O defrag V8.5 will now work in screensaver mode, and if used on a Notebook, defrag will stop automatically when external power is removed, to avoid wasting battery power.

back to top

Windows 2003 Defragmentation Improvements

The main enhancements are -

• A fragmentation process can be scheduled or scripted
• Performance is faster than in Windows 2000
• You can defragment the MFT online. The MFT is an index to all the files on the system including itself, so it needs to be optimised. Disk Defragmenter cannot move the first 16k of the file, as this describes the file system. However it will consolidate the rest of the file.
• You can now defragment any NTFS cluster size
• Defrag works fine on Windows cluster volumes, as long as defrag scheduling is done by virtual node name or cluster name
• Disk Defragmenter will try to place all the files needed at boot time into a single contiguous extent to optimise boot performance.
• You can run reports on the defragmentation status of remote servers with the DefragAnalysis report. This only works when running on a Windows XP workstation, reporting on Windows 2003 servers

back to top

By entering and using this site, you accept the conditions and limitations of use