SAN Physical design: Cabling and Connections

Cable Types

A Fibre Channel SAN can be connected with either fiber or copper cables. Yes, the Fibre Channel standard does support copper. Optical Fiber cables are more expensive than copper, but they work over much longer distances, are a lot more reliable and a lot thinner. The thickness of the cable makes a huge difference if you have a large bundle of them under a machine room floor.

Optical fibers come in two modes; single mode 9 micron and multi mode 50 and 62.5 micron. In general the thinner the cable, the further the light goes. In laymans terms, single mode fiber is thin enough to restrict the light, so that it does not refract, or bounce off the fiber walls. They also use lasers with a longer wavelength than multi-mode. This means that only one wavelength or path is allowed down the fiber. In multi-mode fibres, the light can refract off the walls, so several wavelengths and paths are available.

If you are cabling up new kit to existing cables, then be aware that while you can join 50 micron multi-mode to 62.5 micron multi-mode, you will get some light loss when going from 62.5 to 50. Some of the light will leak away around the edges. If at all possible, you should keep a consistent cable size throughout a path.

Most machine halls use structured cabling these days, where individual fiber pairs are aggregated together into a cable, and terminated on a patch panel. Jumper cables are then used to connect appliances to the patch panel. Every connection will result in some light loss, so there are limits on how many patch panels can be in a circuit. It's best to check with individual vendors for details.

Connections

Fiber cables typically come with two different types of ends, SC or LC. Now in my ignorance, I used to think that they stood for short connector and long connector, which was confusing as the LC connector is actually smaller than an SC. In fact, they stand for Siemens and Lucent. The type of connector you need is determined by the type of socket in your SAN switch or HBA. SC connectors fit into GBICs (Gigabyte interface connectors) while LC connectors plug into SFPs (Small Form Factor Pluggable). SC terminations are generally used for 1Gb/s fibers, and LC for 2Gb/s.
Fiber cables come as a pair of fibers, and each end will have a send and a receive fiber, known as TX and RX. An SC plug will only fit one way into a GBIC socket, and the most common error I've come across when recabling SAN switches is that the fiber pair is crossed. This means that the LED on one side is shining at the LED on the other side, and nothing is working. You will also see that the transmission light is orange instead of green. In this case, it is reasonably easy to unclip the two fibers from the plug and swap them over. However take care not to break the glass fiber itself.
The picture below shows the two common fiber connectors

Other types of connector include HSSDC (High-Speed Serial DataConnection) and DBm/DBf. You may also see MIAs or Media Interface Adaptors. They have a standard DB9 serial plug on one end and an SC socket on the other end. They are used to convert an electrical signal to optical.

Port types

There are two types of device ports, N_PORT and NL_PORT. An N_PORT or Node Port is attached to a fabric, an NL_PORT or Node Loop Port is attached to a Loop.

A SAN switch port can be

• An F_PORT or Fabric Port- essentially the other side on an N_PORT. An F_POT can only connect to one device.
• An FL_PORT or Fabric Loop Port - the other end of an NL_PORT. One FL_PORT can connect to up to 126 devices on the FC-AL loop. An NL Node can be either Public or Private. If it is Public and the FC_AL loop is attasched to a fabric, then the code can communicate with other devices in the fabric. If it is Private then it can only communicate with devices on the loop.
• An E_PORT or Extension Port - used for ISL links, so the other end will be an E_PORT on another switch
• Ports can also be in unassigned status, when they are G_PORTS or U_PORTS

Port Addresses

Every port has a unique address. Some ports have special addresses called well known names. In Brocade terminology, some of these are

• FFFFF6 - The clock server used to synchronise events over the fabric
• FFFFF7 - The security server used to store encryption keys
• FFFFF8 - The alias server
• FFFFFA - The management server
• FFFFFB - The time server
• FFFFFC - The directory server contains details of every node in the fabric. Every node must register with the directory server when logging in, and can query the directory server to get name service address information for other nodes. This is the equivalent of the TCP/IP DNS
• FFFFFD - The fabric controller, this receives state change notifications when a port is taken offline or put back online. The Fabric Controller is in overall charge of the fabric, and so is generally replicated in every switch to ensure no SPOF.
• FFFFFE - The fabric login server or port service, used by devices to establish a connection with the fabric on fabric initialisation, or if a node is re-connected.
• FFFFFF - The broadcast server

A Node is an appliance that is connected to a fabric SAN. Every Node has a unique 64-bit address called 'Node World Wide Name'. Every Port also has a unique 64-bit address called the 'Port World Wide Name'. These addresses are usually writen as a sequence of 8 hex bytes separated by colons like this 10:00:00:60:69:50:60:02. Bytes 3-5 are assigned to each vendor by the IEE naming standards body. Every switch has a unique WWWN node name, and every port in a switch has a unique WWWN port name. Every HBA card has a unique name, and every port on an HBA has a unique name.
This unique set of names means that it is possible to specify exactly which port in which switch or HBA that you need to address, pretty much in the same way you can use a telephone to reach a person.

Buffer Credits

When a data record is transmitted down a Fibre Channel record it is split up into a number of 2148 byte blocks. These blocks are transmitted through the fabric, then re-assembled at the far end to create the data record again. Switches are store-and-forward devices, they do not just pass the data straight through. To make sure the ISL fibers are used efficiently, every switch port has a number of buffers associated with it, called buffer credits or BB_Credits. The switch can then store several blocks of incoming data, while waiting to pass it on to the next node.When a receiver is ready to take information, it signals to its sender, then decrements the BB_Credit. When the data block is passed on to the next receiver the BB_Credit is incremented again. This means that BB_Credits are also used to throttle back the data transmission flow when devices or links get too busy.
EE_Credits or End to End credits are established between two communicating N_Ports and control the overall flow of the data stream through the fabric.
On a Brocade switch, you can check status of these buffers with the command 'portbuffershow'. This will tell you, among other things, how many buffers are allocated to each port, how many are in use, and how many are needed for efficient channel usage.

Record Keeping

Record keeping is the boring, but absolutely essential part of installing and managing a SAN. A good set of records makes it much easier to fix things when they go wrong.
At a minimum, you should have a diagram that shows every port on every switch, what type of port it is and what it is connected to. The good news is that most SAN management packages will produce this documentation for you, but it is important that you keep extract a fresh copy every time you make a change. Export it off onto a laptop, or even print it out, so you can take it into the machine hall with you.

back to top

Copyright © Lascon Storage Ltd. 2000 to present date. By entering and using this site, you accept the conditions and limitations of use