Storage Area Networks - SAN Fabrics

There are two main types of SAN, Fibre Channel and iSCSI. The two are not mutually exclusive and can even be interconnected (A word about spelling. Fibre Channel is a standard, and that is its name. However, apologies to fellow Brits. but I'm using the American spelling of fiber elsewhere to refer to cabling throughout these pages).

Fibre Channel has three main variants, Point to Point, Arbitrated Loop and Switched Fabric. Point to point is just simply a pair of fibers that connect one server to one storage device. While not a SAN, this is an improvement over SCSI as the cable distance can be much longer.

FC-AL

The original SAN type was Fibre Channel, and older fabrics often used FC-AL (Fibre Channel Arbitrated Loop). While FC-AL can support 125 devices on a loop, only two devices can be active on the loop at once, which restricts performance. Many people will quote 127 devices which is technically accurate, but address 00 is reserved for the FL_Port, so in practical terms you can only connect 126 devices.
FC-AL was very suitable for connecting tape drives to servers, and was often implemented as a Hub, a single switch that healed itself if one port became in-operational. FC-AL is still typically used inside storage subsystems to manage the disk strings.

Switched SAN Fabrics

The most common type of Fibre Channel SAN today is a switched fabric. Just to set the scene, here is some common terms that are used when talking about a SAN. A 'Node' is a server or a storage devide. A 'Fabric' is the network that connects these nodes together, and includes the network switches. A 'Domaim' is a single switch within a fabric. A fabric can contain up to 239 domains, and in theory, the fabric can scale to about 16 million connections. Switches can vary in size from small 16 port devices with little or no redundancy, to large 'directors' with up to 728 ports and no single point of failure. There are five main types of fabric:

Single Switch

Single switch is the simplest fabric, but it can vary from a single 16 port switch to connect a few servers and a couple of storage devices, to a large 256 port director that connects a large enterprise together. A small switch is a single point of failure and will not scale, but can be a good way to start out. Is a director with no internal SPOF an SPOF itself? I've heard that one argued both ways. A large switch is also scalable, until it runs out of spare ports.
Single Switch SAN
 

Now a variation on a single switch is two switches (bear with me). If all your servers and storage devices are dual pathed, and you have a multi-pathing failover capability, then you can build a very resilient network with two single switch fabrics. Every device is connected to both fabrics, so if any part of one fabric goes down, all will work fine on the other fabric until the problem is fixed. This is a much better option than a single switch.
Single Switch / Dual Fabric SAN
 

Cascade

In a Cascade SAN, the switches are simply inter-connected in a queue as shown below. There may or may not be a top level switch. An issue with this design is that you do not want to have to go through several hops to connect devices at either end of the queue, so you really need to try to localise your paths through the fabric if possible to go through no more than one ISL, and ideally, connect in the same switch. This makes a cascade design difficult to scale or change later. On the plus side, a cascase design does not need too many ISLs.
Cascade Fabric SAN
 

Loop

Loop Fabric SAN

A Loop fabric is essentially just a cascade fabric with the bottom switches connected together to form a ring. Slightly harder to extend that a cascade, as you need to break to loop to install another switch. Otherwise it has the same drawbacks and benefits as a cascade SAN.

Full Mesh

In a Full Mesh SAN, every switch is connected to every other switch with an ISL. The advantage of this approach is that you can connect any device to any open port in the fabric, and know that it can connect to any other device after just one switch hop. The big disadvantage is scalability. When you add a new switch, it must be connected to every other switch in the fabric. It is obviously not suitable for low port count switches, as then most of the ports are used for ISLs.
Full Mesh SAN
 

Core/Edge

A Core/Edge SAN is a logical progression from full mesh as it does away with the requirement for lots of ISLs while preserving the one switch hop rule. It uses a high performance director for the core switch, which is connected directly to high performance servers and storage. Appliances that need lesser performance are connected to the core by slower edge switches.
Core-Edge SAN
 

Once you start building large SANs with lots of switches, you might not want to go to the expense of having two complete and separate fabrics for failover, but you still want redundancy. A Federate Fabric SAN contain redundant switches, so that every server is connected to two switches, and has two independent paths through the SAN to the storage. Again, the host servers must have multi-pathing software that can automatically failover if a path fails, and ideally load balance when two paths are available.
When a complex fabric is started up, it needs something to make sure that the switches are connected correctly, have unique domain ids, and are time synchronised. This can be done using fabric management software, but the Fibre Channel standard allows for a 'Principal Switch' or master switch to manage the network.

HUBs and Routers

You may still want to connect an FC-AL loop into your SAN through an FL_PORT, as some devices will only support FCAL. However rather than cable up all the devices on the loop point to point, it is better to attach them through a Hub.

Routers or Bridges are used to interface Fibre Channel with SCSI, typically to connect up devices that do not support Fibre Channel, such as older tape libraries. Routers are not needed so much these days, partly due to newer devices being FC capable, and partly due to the advent of iSCSI SANs.

Host Bus Adaptors

Host Bus Adaptors or HBAs are circuit boards that are installed within a server. As the name implies, they are the interface between the fiber cable and the internal server bus. HBAs can support copper or fiber, and usually have two GBIC or SFP external connections. HBAs are expensive relative to the price of a server, which in one reason why iSCSI SANs are becoming popular.

iSCSI SANS

iSCSI SANs were proposed at the end of 2002 and matured into usable products in 2004. The biggest advantage of iSCSI is that it is much cheaper than fibre channel. It also makes clustering easier, supports multi-path IO and makes multi-site replication easier. ISCSI is still considered the optimum choice for moderate to low performance applications, though many people do use it for top class work. It is typically used for Windows, Netware or Linux, with some small Unix takeup. Application servers that are Ethernet enabled can be attached directly to an iSCSI SAN, they do not need expensive HBA cards.
An iSCSI solution provider can be found in iSCSI SAN
iSCSI storage solutions developed by IP SAN Company SANRAD: Secure and manageable, SANRAD's iSCSI and IP SAN solutions allow you to manage your IP storage network easily and effectively.
This site includes some excellent white papers on iSCSI usage.

back to top

Copyright © Lascon Storage Ltd. 2000 to present date. By entering and using this site, you accept the conditions and limitations of use