SAN Physical design: Cabling and Connections

Cable Types

An Optical Fiber consists of a very thin glass core surrounded by a cladding. The cladding has a lower refractive index than the core, so the boundary between core and cladding acts like a mirror and reflects light down the core. The light source is usually infra-red and might not be visible to the human eye. Optical Fibers come in two modes; single mode and multi mode with the thicknes if the light pipe measured in microns. Popular sizes are 9, 50 and 62.5 microns. In general the thinner the cable, the further the light goes. In laymans terms, single mode fiber is thin enough to restrict the light, so that it does not refract, or bounce off the fiber walls. This means that only one light wavelength can go down the pipe, but it suffers less interference and so can travel further without losing effective signal strength. Single mode also uses light emitters with a longer wavelength than multi-mode. In multi-mode fibres, the light can refract off the walls, so several wavelengths and paths are available. Single mode is more expensive than multi-mode and is typically used where transmission speed is essential, for example for multiplexed remote syncronous data mirroring.

If you are cabling up new kit to existing cables, then be aware that while you can join 50 micron multi-mode to 62.5 micron multi-mode, you will get some light loss when going from 62.5 to 50. Some of the light will leak away around the edges. If at all possible, you should keep a consistent cable size throughout a path. 62.5 micron cables are generally not used for new installations and are becoming obsolete.

   

Accelerate DB2 Write with zHyperWrite and "EADM™ by Improving DB2 Logs Volumes Response Time:

The transmission speed and acceptable distance of a fiber depends on a number of factors, including light source and cable characteristics. In general, a single-mode 9 micron cable can trasmit up to 50km and a multi-mode 50 micron cable can transmit up to 900m.
One fiber term you will come across is 'dark fiber', usually used as if it is some special kind of fibre. When utility companies lay fiber in the ground, they always install more than is required for current needs, as the installation is by far the most expensive cost. This spare fiber has no light going down it, so it is 'dark'. And one other thing, I'm English and I like to use correct English spelling. Industry standards are that the architecture is called Fibre Channel and the cables are called fibers, which should explain why I spell Fibre two different ways.

Most machine halls use structured cabling these days, where individual fiber pairs are aggregated together into a cable, and terminated on a patch panel. Jumper cables are then used to connect appliances to the patch panel. Every connection will result in some light loss, so there are limits on how many patch panels can be in a circuit. It's best to check with individual vendors for details.

Connections

The most common types of fiber optic connectors that are available now are SC, LC, ST, FC, and MT-RJ. Most of these types of connectors can be used with either multimode or single mode fiber.
There is some dispute as to what SC stands for. I think it was originally 'Siemens Connector', but the official term now is 'Subscriber Connector'. Other variants are 'Square', 'Small', 'Set and Click' or 'Stab and Click'. SC fits into a GBIC (Gigabyte interface connector) and has a square molded plastic body. To connect it you push it into the GBIC until it locks. It is larger than LC and is generally used for slower speed fibers.
LC stands for 'Lucent Connector' and plugs into SFPs (Small Form Factor Pluggable). LC for are used for faster speed fibers where space is important.
ST stands for Straight Tip and is a quick release bayonet style connector with a twist lock coupling. They are more common than SC connectors.
FC or fiber-optic connector has a threaded body, and is commonly used with single-mode optical fiber. They are becoming less common, displaced by SC and LC connectors.
MT-RJ stands for Mechanical Transfer Registered Jack and is very popular for small form factor devices due to its small size. Housing two fibers and mating together with locating pins on the plug, the MT-RJ comes from the MT connector, which can contain up to 12 fibers.

Other types of connector include HSSDC (High-Speed Serial DataConnection) and DBm/DBf.
When copper cables and copper-based Fibre Channel devices are used in a SAN, they are attached to other Fibre Channel devices using two types of connectors, Copper Gigabit Interface Connectors (GBICs) and Media Interface Adapters (MIAs). Copper GBICs are hot pluggable connectors that attach to Fibre Channel devices using either a DB-9 or the High Speed Serial Data Connector (HSSDC). Media Interface Adaptors have a standard DB9 serial plug on one end and an SC socket on the other end. They are used to convert an electrical signal to optical.

Fiber cables come as a pair of fibers, and each end will have a send and a receive fiber, known as TX and RX. An SC plug will only fit one way into a GBIC socket, and the most common error I've come across when recabling SAN switches is that the fiber pair is crossed. This means that the LED on one side is shining at the LED on the other side, and nothing is working. You will also see that the transmission light is orange instead of green. In this case, it is reasonably easy to unclip the two fibers from the plug and swap them over. However take care not to break the glass fiber itself.
The picture below shows the two common fiber connectors

Fiber connectors

Another parameter you might come across is polish type. There are three types of polishes, 'PC' or Physical contact, 'UPC' or Ultra Physical contact and 'APC' or Angled Physical contact. These are important as each polish type reflects light off the end of a fiber connector in a different way. If the types of polish are not matched correctly then a lot of the light signal can be lost.
APC connectors come with the fiber end face polished at an angle to prevent light that reflects from the interface from traveling back up the fiber. Angle-polished connectors should only be mated to other angle-polished connectors. You can usually identify angle-polished connections as they have either a green strain relief boot or a green connector body. Also, the name of the connector might have '/APC' (angled physical contact) added to the end. Two different versions of FC/APC exist, FC/APC-N (NTT) and FC/APC-R (Reduced). An FC/APC-N connector key will not fit into a FC/APC-R adapter key slot.

Port types

There are several types of device ports:

The following types of ports are also used in Fibre Channel:

Node and Port Addresses

A Node is an appliance that is connected to a fabric SAN and every Node has a unique 64-bit address called 'World Wide Node Name'. Every Port also has a unique 64-bit address called the 'World Wide Port Name'. These addresses are usually writen as a sequence of 8 hex bytes separated by colons like this 10:00:00:60:69:50:60:02. Bytes 3-5 are assigned to each vendor by the IEE naming standards body. Every Node as a unique WWNN node name, and this can mean individual HBAs, switches, storage units, storage arrays, tape libraries and tape drives. Every port on every node has a unique WWWPN port name.
This unique set of names means that it is possible to specify exactly which port in which switch or HBA that you need to address, pretty much in the same way you can use a telephone to reach a person.

Buffer Credits

When a data record is transmitted down a Fibre Channel record it is split up into a number of varying size blocks or frames. These blocks are transmitted through the fabric, then re-assembled at the far end to create the data record again. Switches are store-and-forward devices, they do not just pass the data straight through. To make sure the ISL fibers are used efficiently, every switch port has a number of buffers associated with it, called buffer credits or BB-Credits. The switch can then store several blocks of incoming data, while waiting to pass it on to the next node. When a receiver is ready to take information, it signals to its sender, then decrements the BB-Credit. When the data block is passed on to the next receiver the BB-Credit is incremented again. This means that BB-Credits are also used to throttle back the data transmission flow when devices or links get too busy.
Buffer credits are consumed by frames, not data. A large frame and a small frame will both use one buffer, so it's frame count, not the amount of data that matters.
EE-Credits or End to End credits are established between two communicating N-Ports and control the overall flow of the data stream through the fabric.
On a Brocade switch, you can check status of these buffers with the command 'portbuffershow'. This will tell you, among other things, how many buffers are allocated to each port, how many are in use, and how many are needed for efficient channel usage.

Record Keeping

Record keeping is the boring, but absolutely essential part of installing and managing a SAN. A good set of records makes it much easier to fix things when they go wrong.
At a minimum, you should have a diagram that shows every port on every switch, what type of port it is and what it is connected to. The good news is that most SAN management packages will produce this documentation for you, but it is important that you extract a fresh copy every time you make a change. Export it off onto a laptop, or even print it out, so you can take it into the machine hall with you.

Storage Area Networks

Disk Protocols

Flash Storage

   

Accelerate DB2 Write with zHyperWrite and "EADM™ by Improving DB2 Logs Volumes Response Time:

Lascon latest major updates

Welcome to Lascon Storage. This site provides hints and tips on how to manage your data, strategic advice and news items.

back to top