40- and 100-Gigabit Ethernet Connectivity Solutions

40- and 100-Gigabit Ethernet Connectivity Solutions
By Lisa Huff, Bishop & Associates Inc.

Ethernet has historically increased data rates by 10-times. This new variation marks the departure from this trend. This was done primarily in response to server manufacturers. The addition of the 40G objectives was spearheaded by Sun Microsystems, Intel, and Broadcom. Sun presented data showing server I/O requirements versus time (figure 1, below).

This shows that as each new data rate has been released, the servers’ adoption of it lags by at least three years. And, for 10G it has been more than eight years before the servers have been able to take advantage of the I/O speed. Server manufacturers were concerned that if the IEEE adopted 100G without a “speed bump” in between, that it may drastically slow its adoption. There was a fierce fight in the standards group, but eventually 40G was added, and we see early adoption of it, at least for high-end data center applications, only a year after the standard was released.

40 and 100G Ethernet Networking
The IEEE 802.3ba task group was formed in December 2007. The original objectives of the group were:

Support full-duplex operation only
Preserve the 802.3/Ethernet frame format utilizing the 802.3 MAC
Preserve minimum and maximum frame size of current 802.3 standard
Support a BER better than or equal to 10^-12 at the MAC/PLS service interface
Provide appropriate support for OTN
Support a MAC data rate of 40G
Provide physical layer specifications, which support 40G operation over at least:
10km on SMF
100m on OM3 MMF
10m over a copper cable assembly
1m over a backplane
Support a MAC data rate of 100 Gb/s
Provide physical layer specifications, which support 100 Gb/s operation over at least:
40km on SMF
10km on SMF
100m on OM3 MMF
10m over a copper cable assembly

The initial firms that drove the HSSG were primarily interested in a single-mode solution for 100G Ethernet. Numerous presentations by Cisco, Force10, and HP attempted to create a focused project with a short timeline that would facilitate a standard in one-to-two years.

The addition of the copper objectives was based on input from Lawrence Livermore National Laboratories (LLNL). It requested a copper reach of at least 5m, and preferably 10m. After products were developed, it was clear that there would actually have to be two separate copper distances — 7m for passive assemblies and 15m for active ones. The specification came out with a 7m minimum.

The BER objective of 10^-12 was discussed at length, with a large group requesting a 10^-15objective. In the end, the group decided that because of test issues it was not feasible to specify a 10^-15, but it was agreed that system vendors would require it.

A follow-on study group was formed and just released the IEEE 802.3bg standard for serial 40G on SMF that would be compatible with telecom 40G networks. This variant — the 40GBASE-FR — is expected to eventually replace the current 40GBASE-LR4.

A summary of the 40/100G variants are shown in Table 1, below.

40- and 100-Gigabit Ethernet Variants
Table 1

Except for just a few implementations like Brocade’s switches in the Amsterdam Internet Exchange’s network, 100G Ethernet is still being tested in the lab. But all the top switch manufacturers have released 40G products that are slowly being adopted into data center networks.

Traditionally, Ethernet switches lag the development of server technology when it comes to speed migration. This is because it is much easier to develop a chip set and optical module to fit on a traditional network interface card (NIC) than it is to fit it into a high-density switch port. We saw this with gigabit and 10-gigabit Ethernet, but it seems to be reversed for 40G. Several switch manufacturers have already released products with multiple 40G ports. The switches have 48 10G SFP+ ports for connections to servers and four 40G QSFP+ ports for uplinks to aggregation/distribution switches. Figure 2 (below) shows one.

Connectors for 40 and 100G Ethernet
As seen in Table 1, there has yet to be a twisted-pair/RJ45 solution developed for 40 and 100G. The current implementations include short-reach twinax copper and short-reach (laser-optimized multi-mode), and long-reach (single mode) fiber. For 40G, the form factor is QSFP+ and for 100G it is currently CXP or CFP, but these have some issues that equipment manufacturers do not like. The CXP can only cover 100m and is not compatible with the CFP that is used for longer distances. The CFP is too large to get any kind of port density on the switch. So component suppliers are already working on new form factors for 100G applications even before there is a mass market for them.

QSFP+ Connectors and Cables
The QSFP form factor was originally developed for Fibre Channel director-switch chassis connections at lower data rates. Similar to the SFP, the QSFP has been improved to handle higher data rates, and thus became the QSFP+.

While the CFP is being used for some short-reach connections for 40G Ethernet, the vast majority are expected to be QSFP+ for both copper and fiber. In fact, if end users had their way, the QSFP+ would also be used for 40GBASE-LR4, but transceiver manufacturers have yet to figure out how to fit the needed chips and actually cool them in such a small footprint. However, they are all working on it.

Figures 3 through 5 show the QSFP+ family of products for 40G Ethernet.

CXP and CFP Connectors and Cables for 100G Ethernet
The CFP form factor is used for long-reach applications — beyond 100-meters right now — and the CXP is being used for short-reach copper and fiber. Again, the copper solution is a DAC. The fiber products are available in both an AOC and modules with cable assemblies, but for Ethernet, the separable modules and cable assemblies are what customers seem to want.

Figures 7 through 10 show the CXP family of products. Figures 11 through 14 show the CFP products. Current CFP module implementations use SC optical connectors as its network interface, but LC and possibly MPO connectors are expected to follow.

Bishop & Associates has just finished a study on Gigabit data communications connectors and cable assemblies and sees a bright future for 40/100G Ethernet products. Figure 15 shows a snapshot of this market and is one of the many forecasts put forth in Bishop’s new report “Gigabit Data Communications Connectors and Cable Assemblies, 2011 to 2015.”

Lisa Huff
Telecom Director, Bishop & Associates Inc.
Lisa Huff is a Certified Data Center Professional and electrical engineer with more than 25 years experience in the electronics industry. Her connector and market research-related work includes being a manager in Nexans Inc. North American Competence Center; a marketing manager at Berk-Tek, A Nexans Company; an optical components analyst for Communications Industry Researchers (CIR); a communications marketing manager at FCI; and development engineer at AMP Incorporated (now TE Connectivity). Her expertise is in data centers, data communications cabling and connectivity, networking equipment, and optical components. Lisa has produced more than 20 publications, including market research reports, industry Webinars, articles, short courses, and white papers.