Local Area Networks Transition from Gigabit to 10-Gigabit Ethernet

Local Area Networks Transition from
Gigabit to 10-Gigabit Ethernet
By Lisa Huff, Bishop & Associates Inc.

Almost every enterprise has a data center. In some, it is nothing more than a large wiring closet with a small amount of compute and storage. In others, like Facebook, Google, Yahoo, and Microsoft, multiple buildings warehouse these functions. There are many more of the former than the latter, but most analysts focus on the large Internet Data Centers (IDCs), mainly because these are the early adopters of new technology. However, it is important to also watch current market transitions, because these are what will allow the new technology to be developed. The sustained growth of more well established networking products like gigabit and 10-gigabit Ethernet encourage components companies to invest in these future technologies.

Data Center Defined
A data center is a room (or building) that houses most of a company’s servers and storage. Some also have telecommunications equipment. Ethernet connectivity and cabling used in these rooms has changed over the years. It started with point-to-point connections, but as the number of devices grew, it has evolved into structured cabling. And now, it may be transitioning to what some would call a “hybrid” solution of the two.

Copper Structured Cabling and Connectors
Structured cabling is a building or campus telecommunication cabling infrastructure that consists of a number of standardized smaller elements (hence structured) called subsystems. It is governed by TIA-568 specification for a building and TIA-942 for data centers. Figure 1 below shows a schematic of a basic data center structured cabling infrastructure.

Copper connectors used in this scenario include RJ-45 (modular jacks) for switching and server equipment, RJ-45 plugs (as part of a patch cable), and RJ-45 IDC connector patch panels. The copper cabling that connects them consists of two types, horizontal and patch. Depending on the cable grade, for up to 10G the reach can be 100 meters total. Figure 2 shows a Category 6 installation in a 25,000-square-foot enterprise data center.

Figures 3 through 5 show the individual connectivity products.

Interconnect Cabling
While interconnect cabling has always been available for these high-speed connections, it has changed its form. Instead of having different separable connectors than either the copper structured cabling or fiber, it now uses the same form factor as fiber. These assemblies are known as direct-attach copper (DAC) cables because the cable is directly attached to a PCB internal to the assembly that then plugs into an internal connector in the active equipment. So if one buys a switch with an SFP+ port on it, this copper solution can be used to connect within the rack and the SFP+ optical module can be used to connect outside the rack. Consequently, you eliminate the problem of filling up pathways and spaces with large, bulky category cabling and give flexibility to your switch port distances.

The interconnect cabling solutions at gigabit and 10-gigabit are essentially the same. They use the SFP and SFP+ form factors as shown in Figures 6. The SFP has been replaced by the SFP+ that can accommodate both 1G and 10G connections. The SFP was only designed for applications with data rates up to about 5G.

Another reason that DACs have gained ground in the data center at 10G is because it has taken switch manufacturers a long time to develop their 10GBASE-T solutions — at least, those that use Category 6A (or 7) cabling. Due to the high power consumption (initially about 10W) of the 10GBASE-T chips, switch manufacturers were unable to develop products with more than two ports, which was too low-density for the data center. Now that the chip’s power consumption is 3W or less, the switch-port density is high enough to justify these products and they are available from multiple vendors. This may enable a renewed effort to deploy 10GBASE-T instead of SFP+ switch ports, but this remains to be seen. Since the SFP+ port still has the advantage of much lower power consumption — usually less than 1W for optical and negligible for DACs — it may still maintain its position as the interconnect of choice.

Evolving Network Architectures
Because of its limited distances (up to 7m), DAC products lend themselves to the new networking topology being adopted in the data center, which includes top-of-rack (ToR) switching. One of the main reasons to adopt ToR is to implement CLOS networking. CLOS networking is a multi-stage network whose main advantage is that it takes less cross-points to produce a non-blocking structure. It is difficult and it can be more costly to implement a non-blocking network without CLOS, so many networking professionals are starting to employ it. Non-blocking architectures are needed because in order to save money and compute resources, many applications are being collapsed onto one server through virtualization. Each application may need to be accessed simultaneously, so the traditional network that is aggregated (and may block the right to use each application at the same time) is not sufficient. Figure 7 shows the current network architecture. With ToR and CLOS, the access layer and aggregation layer are starting to be combined, which also necessitates increased data rates as shown.

Fiber Solutions in the Data Center
With ToR switching and CLOS architecture adoption, fiber is becoming more prevalent in the data center. Getting the fiber closer to the end devices not only enables non-blocking structures but also lower latency.

For 10G and below, the fiber options have been the same for many years. But, just like the game at 10G changed for copper, it has also changed for fiber. If data center managers want to try and leverage their OM1 (FDDI-grade) or OM2 (500 MHz∙km) multi-mode fiber (MMF) cable, it will reach only about 30m. So it is recommended to install at least OM3 (2,000 MHz∙km) laser-optimized MMF and single mode fiber for 10G. For 40/100G, the situation becomes even a little more complicated because the transceivers have now changed from serial to parallel, so what was a 2-fiber channel will now become an 8 or 20-fiber channel. Therefore, many recommend installing pre-terminated MPO solutions for 10G that can be reused at 40 and 100G.

Figures 8 and 9 show the 10-gigabit fiber solutions for the data center.

Bishop & Associates Comments:
Bishop & Associates sees an increase in 10G ports in the coming years. This transition is detailed in the graph below.

As a direct result, there will be significant opportunities for connectivity companies that produce the following products:

Copper
Category 6A, 7 – jacks, plugs, patch panels, patch cords, horizontal cabling
DACs
Twinax cable
SFP and SFP+ host board connectors
Fiber
LC-to-LC MM and SM patch cords, adapters, and enclosures
MPO pre-terminated assemblies, MPO adapters, cassettes, and enclosures
Laser-optimized multi-mode fiber (LOMF) cable
Single-mode fiber (SMF) cable

Lisa Huff
Telecom Director, Bishop & Associates Inc.
Lisa Huff is a Certified Data Center Professional and electrical engineer with more than 25 years experience in the electronics industry. Her connector and market research-related work includes being a manager in Nexans Inc. North American Competence Center; a marketing manager at Berk-Tek, A Nexans Company; an optical components analyst for Communications Industry Researchers (CIR); a communications marketing manager at FCI; and development engineer at AMP Incorporated (now TE Connectivity). Her expertise is in data centers, data communications cabling and connectivity, networking equipment, and optical components. Lisa has produced more than 20 publications, including market research reports, industry Webinars, articles, short courses, and white papers.