An enterprise subsystem can be defined as one that supports all the major operating systems; z/OS, Unix variants, Linux variants, Windows and Netware. The major enterprise vendors are discussed below.
The first section discusses the enterprise products from the five big enterprise vendors; EMC, HDS, IBM, SUN and HP. The second section is a table that compares their products.
EMC
History
EMC started out producing cache memory and developed solid state disks, memory devices that emulated spinning disks, but with much faster performance. These solid state disks were usually re-badged and sold by StorageTek.
Around 1988, EMC entered the storage market in its own name, selling symmetrix disk subsystems with what at that time was a very large, 256MB cache fronting 24GB of RAID 1 storage. Their mosaic architecture was the first to map IBM CKD mainframe disk format to standard FBA open system backend disks, and as such, could claim to be the first big user of storage virtualisation. In those days, EMC developed a reputation for delivering best performance, but at a price.
EMC introduced their latest addition to the symmetrix range, the V-MAX, in April 2009. This is the successor to the DMX-4, so is it just coincidence that 'V' is the Roman number for 5, as well as standing for Virtual?
Architecture
The DMX series uses the Direct Matrix architecture, now called Enginuity. The principle behind Direct Matrix is that all IO comes into the box the front-end directors. These are connected to global memory cache modules, which are in turn connected to back-end directors that drive the IO down to the physical disks. This connectivity is all done by a directly connected, point-to-point fibre-channel matrix. There is no switching or bus arbitration involved. Each DMX-4 data path runs at 4GB/s, and there are 128 concurrent data paths. In 2008, EMC became the first to use flash storage in an enterprise subsystem, for high performance applications.
The V-MAX architecture builds on the older DMX architecture. The first release of V-MAX consists of between 1 and 8 engines. Each engine is built from commodity processors, cache, host adapters and disk adapters. This makes them relatively cheap to produce and easier to upgrade later. Internally, the engine components communicate locally, so memory access is local. However, the engines must also communicate with each other and also support the Enginuity global memory concept. To achieve this, the memory is virtualised, and each engine communicates with other engines using fiber connect and RAPIDIO technology. When a director gets a memory request it then checks the location, and if it is local it is served at memory bus speeds. If it is remote, then the request is packaged up and sent off to the remote director for processing. Presumably EMC have optimised this setup to ensure that most memory accesses are local. Certainly the EMC diagrams show each engine with 2 directors, 16 host ports and 16 disk ports, but only 4 virtual matrix ports. There are two of these ports per director, and they are connected to other engines with two MIBE (Matrix Interface Boards). The Cache memory is mirrored, and in configurations with 2 or more engines, it is mirrored between engines.
This architecture extends the direct matrix principle, but now the matrix is virtual. One of the difficulties in machine hall design is leaving room for various frames to grow as cabinets are added to increase capacity. The plan is that V-MAX frames do not need to be in the same footprint, they can be 100m apart.
One interesting feature is the storage tiering, based on T0 Flash storage, T1 FC drives and T2 SATA drives. A new FAST (Fully Automated Storage Tiering) product will automate movement between tiers. Again, the devil will be in the detail. Will FAST move data at file level, block level or LUN level? Will it be driven by business policies?
Models
The Symmetrix DMX-4 became generally available in August 2007. It is very scalable, from 240-2,400 drives. It supports 1 TB SATAII drives and 73 or 146GB flash drives. Internally, it uses 4Gb/s communications end to end, with 4 Gb/s support for FICON or Fibre Channel host connections, internal connectivity and Fibre Channel Drives. The backend architecture is FCAL.
The V-MAX starts with the single-cabinet, entry-level Symmetrix V-MAX SE that can hold 120 disks. This can be extended by adding up to 10 more frames, each holding 240 disks. Note that in this first release, all the frames must be adjacent to each other.
Software
DMX software includes EMC Symmetrix Management Console for defining and provisioning volumes and managing replication. The Time Finder products are used for in-subsystem and PIT replication, and SRDF for remote replication. SRDF can run in full PPRC compatibility mode, and can also replicate to three sites in a star configuration.
Enginuity 5784 adds new features including SRDF/EDP (Extended Distance Protection) which is similar to cascaded SRDF except that it uses a DLDEV (DiskLess Device) for the intermediate hop.
EMC was lacking in z/OS support for some years, but they have recently licensed PAV and MA software from IBM, and have provided z/OS Storage Manager to manage mainframe volumes, datasets and replication.
Openness
The DMX in general is not an Open implementation. SRDF, for example, will only work between EMC devices, and even then, not with all of them. EMC Open Replicator has the ability to take PIT copies from selected non-EMC subsystems to DMX, or to copy from DMX to selected non-EMC devices.
The V-MAX is a closed virtual system, as it cannot connect to storage subsystems within the EMC range.
Full FAST capability is planned for 2010, and full intersite connectivity is also planned, permitting a geographically dispersed storage subsystem, but no date is announced yet.
HDS
History
Hitachi Data Systems was always known as the company that manufactured disks that were exactly compatible with IBM, but worked a little faster and cost a little less. HDS broke that mould when they introduced the 'Lightning' range of subsystems in 2000, which was a merging of telephony cross-bar technology and storage subsystem technology. They extended and developed that architecture further with the USP (Universal Storage Platform), released in September 2004.
Architecture
The USP architecture is based on cross-bar switch connectivity, in-subsystem virtualisation, the ability to partition a subsystem into component LPARS and to ability to replicate data to externally attached subsystems.
The USP components are Disk Adaptor modules, Front End Director modules and Cache modules. These are connected by a set of massively parallel cross-bar switches called a Hierarchical Star Network (HSN). The 'massively parallel' bit means that each switch has a dedicated fibre link to every input and output component, so the switch can set up multiple and parallel paths between devices. This makes the architecture non-blocking. The USP also has two HSNs, one for the physical data, and one for the control data.
The USP supports subsystem based virtualisation, with the ability to attach OEM devices behind the subsystem and virtualise them. This means that the USP can replicate and transparently migrate data internally, and also to externally attached subsystems
The USP can also be partitioned into 32 LPARs or Virtual Private Storage Machines which can be used to isolate workloads. These LPARS can consist of a mixture of internal USP storage and externally attached storage. It is not possible to replicate data between LPARS, and all z/OS data must be kept in the base LPAR.
Models
The enterprise USP comes in four models, with an entry level NSC55 device. Hardware wise, the newer USPV looks the same as a USP1100. The USP
models are all cabinet mounted and differ in maximum cache size, number of array frames and so number of internal disks supported and number of cross-bar switches. The older USP devices can connect up to 32 Petabytes of external storage, at least in theory, while the USPV can connect up to 247 PB externally. It is possible to upgrade between USP models without requiring an outage.
Model
Frames
Cache size
Internal disk drives
Internal raw capacity
Number of cross-bar switches
USP100
1+1
64 GB
256
76 TB
2
USP600
1+2
128 GB
512
153 TB
4
USP V
1+4
512 GB
1,152
2.269 TB
8
USP VM
1+1 racks
472 TB
240
72 TB
1
Software
The USP uses HiCommand to manage both disk and virtualisation configuration.
Replication software is provided under an umbrella product called Business Continuity Manager. This is integrated into the HiCommand product which means that z/OS and Open Systems replication can all be managed and monitored from one point. The actual replication software includes ShadowImage (FlashCopy compatible) for in-subsystem replication, TrueCopy for remote synchronous and asynchronous replication and Universal Replicator for remote asynchronous replication.
Openness
The USP is an open architecture, in that it works with disks from many other vendors and virtualises the data. The list of supported vendors includes EMC, HP, IBM and SUN, as well as older HDS devices. In general, the USP will support the hardware, but replaces the OEM replication software with its own.
IBM
History
The original IBM hard drive, the RAMAC 350, was manufactured in 1956, had a 24 inch (609mm) platter, and held 5 MB. The subsystem also weighed about 1 ton. That was a bit before my time, but when I joined IT 23 years ago, the storage market was dominated by IBM, the mainframe was king, and the standard disk type was the IBM 3380 model K which contained 1.89 GB. IBM lost their market leader position to EMC sometime in the 1990s.
IBM introduced their latest subsystem family, the DSxxxx series, in late 2004 in response to competition from EMC and HDS. They updated their internal bus architecture to increase the internal transfer speed by 200% plus over the ESxxx series, and also abandoned their SSA disk architecture for a switched FC-AL standard. The DS8300 is essentially a follow-on from the ESS disk series, and re-uses much of the ESS microcode.
Architecture
The DS8000 architecture effectively consists of two processor complexes called servers that are connected to hosts using host adaptors, and disks using device adaptors.
The processor complex consists of 2-way or 4-way p-570 servers containing two types of cache, volatile and persistent memory. Every write IO is written to volatile memory in one processor, and non-volatile in the other before the write is acknowledged as complete. The subsystem effectively works internally as two separate units, but one server can run the whole subsystem if the other fails.
If the processor complexes hold 4 way power5+ servers then the DS8K subsystem can also be split logically into two completely independent LPARS, either as a 50/50 or a 75/25 split. Each processor complex is split into two server LPARS, and then a Storage Facility Image or SFI is built using one server LPAR from each processor complex. An SFI is sometimes called a storage LPAR, but note that this is not the same as a server LPAR.
Internal connectivity between servers and device adaptors uses RI0-G connectors, the same as is used internally in the p-series servers. These links can run at a 2 GB per second sustained bandwidth and permit the sharing of host adapters between servers. Host adaptors can be either ESCON, or 4Gb/s FICON / FC. Each Host adaptor provides 4 Host connections, but has only two internal protocol engines, so there may be a degree of blocking.
Device adaptors are installed in pairs, and include the RAID controllers. The device adaptors are connected to the disks using Switched FC-AL, but they are not in an FC-AL loop. The disks are allocated FC-AL addresses to allow the switching to work, but once the connection is made, communications are point-to-point Fibre Channel.
For more detail, try http://www.redbooks.ibm.com/redpieces/pdfs/sg246786.pdf
Models
The top range DS8300 is a cabinet mounted subsystem that can support a maximum of 1024 disk drives and will hold a maximum of 460 TB raw capacity using 500GB FATA disks, or 192 TB using 300GB FC disks. The base cabinet holds 128 disk drives, up to two expansion cabinets can be added, each holding 256 disk drives. The raw disks are supplied in blocks of sixteen, but are configured in groups of eight, with each group being called an array group. All the disks in an array group must have identical size and rotation speed. The DS8300 uses two four-way p-570 servers
The DS8100 spec. is similar to the DS8300, except that it will only support one expansion frame, contains a maximum of 384 drives, and has a maximum raw capacity of 192 TB. The internal processor specification is also lower, at two dual processor p-570 servers.
The DS8000 series currently has three Turbo models available: the DS8100 Turbo Model 931, and the DS8300 Turbo Models 932 and 9B2. The 931 and 932 models do not support LPARS, and have 2-way or 4-way processors respectively. The 9B2 model does support LPARS and has 4-way processors.
The DS6800 is rack mounted, and supports up to 16 disk drives in the base unit. With seven expansion units, it can hold up to 128 drives, giving a maximum raw capacity of 38.4 TB with 300GB drives.
Software
The DS software includes Flashcopy for internal subsystem point-in-time data copies, IBM Total Storage DS Manager for configuration and Metro/Global mirror for continuous inter-subsystem data replication.
The older ESS subsystems supported two kinds of z/OS Flashcopy, a basic version that just copied disks, and an advanced version that copied disks and files. DS only supports the advanced Flashcopy.
Flashcopy versions include;
multi-relationship, will support up to 12 targets;
Incremental, can refresh an old Flashcopy to bring the data to a new point-in-time without needing to recopy unchanged data;
Remote Mirror Flashcopy, permits dataset flash operations to a primary mirrored disk; Inband Flashcopy commands, permits the transmission of flashcopy commands to a remote site through a Metro Mirror link;
Consistency Groups, flash a group of volumes to a consistent point-in-time. A consistency group can span multiple disk subsystems.
Remote mirroring versions include;
Metro Mirror, synchronous remote mirroring up to 300km, was PPRC;
Global Copy, asynchronous remote data copy intended for data migration or backup, was PPRC-XD;
Global Mirror, asynchronous remote mirroring;
Metro/Global Mirror, three site remote replication, two sites being synchronous and the third asynchronous;
z/OS Global Mirror, z/OS host based asynchronous remote mirror, was called XRC;
Z/OS Metro/Global Mirror, three site remote replication, two sites being synchronous and quite close together, the third asynchronous and remote.
Openness
The DS subsystem series is self contained and does not interface with any other vendor's storage subsystem. For Open Systems data, IBM does support mirroring and copying to other vendor's subsystems if they are fronted with SVC virtualisation.
Futures
IBM promised a lot of enhancements to the DS series when they were first announced, and has delivered SATA drive support, space efficient Flashcopy, 4Gb FICON and virtual LUN space support so far.
The DS Subsystem LPARing is currently restricted to two LPARS both of which must be the same size. IBM propose to increase the number of LPARs, allow them to be different sizes and allow them to support different microcode levels in future
HP
History
I've always viewed HP as a major Intel player, but not a supplier with much of a presence in the mainframe market. I'm willing to change that
view if someone can share their experiences. HP has been a Hitachi reseller for some time, but while they buy USP hardware from Hitachi, HP supplies its own software.
Architecture
Because the HP XP24000 is a re-badged Hitachi USP, it has the same basic architecture. Older HP devices supported an extra CHIP set to provide more connectivity, but the latest models seems to use standard HDS hardware.
Models
HP now supplies two USP type models, the XP24000 and the XP20000, which are the equivalents of the USP V and the USP VM.
Software
For mainframe solutions, HP seems to supply standard HDS software. For Open Systems solutions, HP supplies its own software. This includes
Storageworks Continuous Access which provides synchronous data mirroring between subsystems
Storageworks Business Copy which provides full volume copy within the subsystem. This looks similar to EMC Timefinder rather than IBM FlashCopy
Storageworks Virtualization system, an internal and external virtualisation manager, can be used for data migration and replication
Storageworks LUN configuration and Security manager which is used to configure the XP12000, to define paths, array groups, volumes and LUNs
StorageWorks Performance Advisor which monitors performance within the XP subsystem
Openness
The XP24000 has the same open architecture as the HDS USP and supports the same range of OEM devices, plus it supports HP MSA devices.
Oracle / SUN
History
Until recently, I've always considered SUN to be a midrange company that supplied storage for its server devices. While they have been Hitachi resellers for some time, they were never big in the enterprise market. However, SUN bought out StorageTek in June 2005 and StorageTek was an enterprise player. StorageTek was the first major company to put solid state cache in front of disks to make them perform better. In the late 1990s they brought out the Iceberg product. From an architectural perspective this box should have cleaned up the market with its innovative design and data management facilities. However even with IBM backing it was unable to really compete with the EMC 8xxx series and HDS 77xx series.
For the enterprise market, SUN no longer sells the Iceberg based Flexline STK V2X devices but now just acts as an HDS reseller, offering the Hitachi USP as a re-badged SUN device complete with standard HDS software. Oracle completed its takeover of SUN in early 2010 and it remains to be seen how much emphasis they will put on their storage arm.
Architecture
As it is the same device, the SUN 9990 architecture is identical to the HDS USP. SUN is not mentioned in the comparison table below, but will have the same data as HDS.
Models
SUN has opted to resell the USP range as a single product, called the SUN Storedge 9990. They resell the NSC55 as the SUN Storedge 9985
Software
The SUN Storedge 9990 runs the same HDS software as the USP
Openness
As the SUN 9990 is a re-badged HDS, it shares the same open architecture.
Netapps
NetApps is becoming one of the major Open Systems vendors. They specialise in the Intel market (Windows, Netware, Linux) and should be considered 'one to watch', rather than a current enterprise storage contender.
Their hardware range includes the Network Appliance series, of which the FAS6070 is expandable up to 500TB.
Their software range includes various SnapMirror and SnapClone products for inter subsystem and intra subsystem data copying. They have MSSQL, Oracle and Exchange agents.
Storage Subsystem Features table
The various suppliers of mainframe disks are contrasted in the tables below. The first row explains why the factor might be important, the second row just presents the facts, which were correct at time of writing, January 2009. However I'd advise you to check with your salesperson for up to date details.
Manufacturer
IBM
EMC
HDS
HP
Device
DS8700
V-MAX
Platform V
(SUN SE9990)
Platform VM
XP24000
Architecture
See the previous page for an
explanation of the various types of disk architecture
PCI BUS
Virtual Matrix
Switch
Switch
Maximum, and maximum effective capacity
How much data can you cram into the box? The maximum
configured capacity will be less than the rated capacity, partly due
to RAID overhead, and partly due to 3390 emulation overhead. The configured
figures are for Mainframe emulation, Open Systems emulation will be
higher. The maximum EFFECTIVE capacity for a mainframe workload running
IO intensive TP systems can be as little as 33% of the maximum capacity,
if you want adequate performance.
450 TB with 450GB FC disks
1 PB with 1TB SATA disks (raw)
Using 2,400*1 TB drives, 2.4 PB raw or 1,967TB usable with RAID7+1 on z/OS
Using 2,400*146 GB drives, 350TB raw or 287TB usable with RAID7+1 on Open Systems.
2,269B raw maximum, RAID6 usable capacities
1,690 TB; Open Systems
796 TB; z/OS
472TB with 2TB SATA drives, 331TB usable with RAID6
2.26 PB raw maximum, 1,980 TB usable
Connectivity
What kind of cables you can plug into the box. A good
box will support both FICON and Fibre at 4Gb/s and maybe 8Gb/s Ethernet. All boxes should support ESCON and iSCSI.
The number of channels available is also important as that determines the overall throughput.
218 4Gb FICON or FC,
64 ESCON
Up to 128*4Gb FICON
128*4Gb Fibre
64 iSCSI
64 Gb Ethernet
112 ESCON
112 FICON
224 FC with 1024 virtual channels per physical port
Up to 64 channels in total, a combination of 48 ESCON, 32 FICON or 64 Fibre
224 FC (4Gb) or 112 FC (8Gb)
112 FICON
112 ESCON
32 iSCSI
Internal Bandwidth
How fast can data move inside the box? The numbers
quoted are marketing figures, you won't really see these numbers in
practice. See the Architecture section for
more information.
64 Gb/s
192 Gb/s with 8 engines
106 Gb/s
13.3 Gb/s
106 Gb/s
Disk Connectivity
See the previous page for
details of disk connectivity.
Switched FC-AL
FC-AL 4Gb 2 port FC
FC-AL
FC-AL
FC-AL
Cache size
In theory, the bigger the cache, the better the performance,
as you will get a better read-hit ratio, and big writes should not
flood the cache. If the cache is segmented, it is more resilient,
and has more data paths through it
32-384 GB
1TB with 8 engines
512 GB
192 concurrent control cache operations
64 concurrent data cache operations
128 GB 96 concurrent operations, mirrored
512 GB
192 concurrent control cache operations
64 concurrent data cache operations
3380/90 emulation
3380 drives are older legacy technology and most sites have now converted to 3390. 3390 comes in multiple sizes, a 3390-3 will hold 2.8 GB. The newest model is the 3390-M.
All models, including EAV
All models, including EAV
All models, supports up to 65,536 logical devices (the older USPs just support 16,384 Open Systems devices)
All models
All models, supports up to 65,536 logical devices (or 16,384 Open Systems devices)
How big are the real, spinning disks and how fast do they run. The bigger the
disks, the less you pay for a terabyte, but bigger disks might be
performance bottlenecks. If you have really large disks, then there
should be fewer of them on an FC-AL loop. Faster speeds less rotational delay.
146, 300 GB and 450 GB FC; all 15,000 rpm
1TB SATA
146,300,450,600 GB FC; 15,000 rpm
1TB, 2TB SATA 7,200 rpm
146,300,450,600 GB FC; 15,000 rpm
1TB, 2TB SATA 7,200 rpm
146,300,450,600 GB FC; 15,000 rpm
1TB, 2TB SATA 7,200 rpm
Flash Disk support
Does the subsystem support Flash disks? They can used
for data that requires very fast access
73GB or 146GB SS
73GB or 146GB SS
200GB or 400GB
200GB or 400GB
73GB, 200GB or 400GB
remote copy
Do you mirror data between two sites? If so you need
this. There are basically two flavours of mirroring, SRDF from EMC,
and PPRC from IBM. SRDF is arguably the better solution technically,
but it locks you into EMC disks. PPRC is used by everyone. The
remote mirroring section has more details.
Global Mirror, asynchronous
Metro Mirror (PPRC), synchronous
Synchronous(SRDF/S) and asynchronous(SRDF/A) data replication between subsystems.
SRDF/DM will migrate data between subsystems.
SRDF/AR works with TimeFinder to create remote data replicas.
SRDF products are all EMC to EMC
SRDF can emulate Metro mirror and Global mirror
'Instant Copy' of volumes or datasets. Can be used
for instant backups, or to create test data. Some implementations
require a complete new disk, and so double the storage. Some implementations
work on pointers, and just need a little more storage.
Timefinder at volume or dataset level. BCV version
requires a complete volume be supplied, newer 'snap' version just uses pointers.
EMC Compatible Flash (FlashCopy)
Shadow Image at volume level
Copy on write snapshot
Storageworks Business Copy, full volume copy, looks similar to EMC Timefinder.
The FlashCopy version provides mainframe copies.
PAV and MA support
Parallel Access Volume and Multiple Allegiance. See
the implementation tips section for details.
Used to permit multi-tasking to logical devices
Yes
Yes , including HyperPAV support
Yes
Yes
Yes (Hitachi PAV)
LPAR Capable
Can the storage subsystem be split logically, so it
appears to be several separate systems, perhaps running different
levels of microcode?
No, but older models do support 2 LPARS with a 50/50 or 75/25 split.
No
Yes (32 LPARS, Z/OS data in single LPAR only)
No
Yes (32 LPARS, Z/OS data in single LPAR only)
Storage Virtualisation Server
Can the storage subsystem act as a virtualisation engine
in conjunction with a SAN? This enables lots of disparate storage
to be controlled from one central point, including mirroring
between different vendor's devices.
No
No
Yes
No
Yes
Manufacturer
IBM
EMC
HDS
HP
Device
DS8700
V-MAX
USP V
SUN SE9990
USP VM
XP12000
Price is usually very negotiable, but be sure to make sure that the vendor quotes for a complete solution with no hidden extras. Also, make sure that you get capped capacity upgrade prices, including increased software charges as software is usually charged by capacity tiers.