QSFP-DD Strategies Help Data Centers Keep Cool
The Quad Small Form Factor Pluggable Double Density (QSFP-DD) is the industry’s smallest 400GbE module and provides the highest port bandwidth density. Rigorous tests verify its suitability as a next-generation high-density, high-speed pluggable module.
The thermal performance of the Quad Small Form Factor Pluggable Double Density (QSFP-DD) module has been extensively evaluated for use in a high-performance data center environment. Thermal test data and results for the feasibility of 15W QSFP-DD modules have been presented and analyzed showing temperature rise versus airflow.
The QSFP-DD is the industry’s smallest 400GbE (Gigabit Ethernet) module and provides the highest port bandwidth density. Developed in collaboration with the Quad Small Form Factor Pluggable Double Density (QSFP-DD) Multi Source Agreement (MSA) group, the QSFP-DD meets market demand for a next-generation high-density, high-speed pluggable module.
Development of the QSFP-DD form factor leveraged the industry’s significant manufacturing capabilities and cost structure supporting QSFP+ and QSFP28 de facto standards for 40GbE and 100GbE. The form factor enables 36 ports of 400GbE in a single rack unit (RU) and provides more than 14Tb/s of bandwidth.
QSFP-DD supports 36 ports in 1RU
The QSFP-DD module is backward compatible with all QSFP-based transceivers from 40Gb/s to 200Gb/s, and can support a range of products including:
- 3m of passive copper cables
- 100m over parallel multi-mode fiber
- 500m over parallel single-mode fiber
- 2km and 10km over duplex single-mode fiber
- Wavelength-division multiplexing (WDM) and coherent optics
Thermal Requirements & Testing
Among the challenges of designing a piece of equipment with pluggable modules is that each socket must be capable of handling the maximum thermal load. Based on a survey of all anticipated optical module types and reaches, cooling of at least 15W is necessary to support the maximum reach of the QSFP-DD.
Fortunately, there is significant industry experience in both building and cooling the small form-factor family of modules, including smaller SFP single-lane modules and backward compatible QSFP four-lane modules, both of which are in densely employed in today’s network switches. These past experiences are translatable to the QSFP-DD pluggable module and, with further innovation, 15W is considered easily achievable in 400GbE products.
1RU port density
System design flexibility to optimize thermal cooling is one of the key advantages of QSFP-DD modules. Their flat top design enables riding heat sinks and/or heat pipes to be optimized with numerous thermal innovations, including range-of-port inlet, port exhaust, and side-to-side cooling options.
High-density systems utilize different printed circuit board (PCB) layouts, fan placement, and air flow control to allow optimization of routing, module placement, and airflow. A belly-to-belly layout may place QSFP-DD modules on opposite sides of the PCB. In this design, the air flows on both sides of the PCB to provide module cooling advantages. This design also provides better signal integrity of the high-speed traces into the modules when compared to a stacked card cage. One downside of the belly-to-belly layout is the limited height of components on the PCB and the reduced height of the heat sink on the high-power switch chip.
Another option, the stacked layout, places QSFP-DD modules on the same side of the PCB, so air flows on only one side. By maximizing the heat sink height, this layout provides advantages for cooling the switch chip. The primary challenge of using the stacked design is maintaining the signal integrity of the high-speed traces to the upper stacked card cage and cooling the lower cage module.
Thermal tests have been conducted to characterize QSFP-DD module and cage thermal performance under specified operating ranges in order to ensure that QSFP-DD solutions remain robust when subjected to temperature extremes. Results of extensive testing have documented detailed airflow and thermal results using T-rise as the key parameter for system design. In each case, the target thermal performance is to hold the rise in temperature from the ambient to the module case below 30°C.
Thermal Test 1: Stacked Cage Test Case
The ability to cool modules in both upper and lower slots requires heat sinks integrated into a 2×1 card cage. Testing was performed to determine the thermal performance of the module-cage and heat-sink combination with high-power optical modules. Module thermal testing was performed using side-by-side 2×1 cages representing a 1RU switch.
Significant thermal testing focus has been on the fixed 1RU system designs, as they typically are the most challenging from a thermal design perspective. Fan space is limited and this form factor represents the most complex module thermal design. Modular system designs with line cards that pull out typically have larger fans and are able to provide higher airflow across the components. T-rise is typically 5–7°C lower in a modular system than in a fixed design.
Two 2×1 QSFP-DD cages were set side by side in each test. All thermal tests were conducted under room temperature, ranging from 20°C to 22°C at sea level. The airflow direction was front to back, and airflow ranges used in the tests were considered typical for system designs. In some test cases, a force gauge/cell was used to set the heat sink downforce at a specified value in order to reach consistent test results.
For an in-depth look a the research behind QSFP-DD cooling solutions for data center environments, visit the white paper section on Molex’s profile page on Connector Supplier.
In stacked cage thermal testing, the temperature rises with clip usage, whether preset at low or preset at high. This resulted in very close temperature results, which is an indication of proper clip force from the clip design. The module average case temperature rise was around 21–22°C when the airflow per 2×1 cage was around 7 CFM. The temperature rise plots indicated that the module case temperature rise can be less than 20°C if the airflow is more than 8 CFM per 2×1 cage. In most cases, the bottom modules in 2×1 cages will run 2–4°C hotter than top modules under the CFM ranges tested. The T-rise versus power graph confirms the ability of the QSFP-DD module and cage combination to support the required 15W with less than 30°C temperature rise.
Thermal Test 2: Belly-to-Belly Test Case
The belly-to-belly design uses heat sinks mounted on the top of surface mount cages and should provide optimum module airflow in a 1RU switch design. In this component-level test there were two 1×2 QSFP-DD cages set on either side of a test board. All thermal tests were conducted at 40°C (114.8°F) ambient temperature. The air flow direction was front to back, and the air flow ranges used in the tests were considered typical for system designs. Belly-to-belly thermal testing was performed with both 14W and 15W module power dissipation. Results show that biasing the power dissipation to the center and rear of the module can result in significantly improved thermal performance.
In a belly-to-belly system with front-to-back airflow, the case temperature of a 15W module can be kept below 70°C (158°F) in a 46°C ambient environment with 6.4 CFM per module airflow. At 40°C (114.8°F) ambient air temperature, the maximum module power dissipation increases to 18W. Optimization of the thermal environment and/or custom heat sinks (e.g., heat sinks with taller fins and/or a higher fin density) can increase maximum module power dissipation to beyond 18W. A 6.4 CFM airflow is required to achieve the desired thermal performance. This is achievable with counter rotating fans at 2.5” H2O pressure drop.
QSFP-DD Modules are Flexible and Economical
Building and sustaining the pipeline of interoperable interconnect solutions is absolutely critical to support advances in transceiver modules, switch technologies, and servers. As data centers maximize the capacity of power and cooling system limits, thermal management becomes increasingly important.
The thermal performance of QSFP-DD modules has been extensively evaluated for use in high-performance data center environments. Resulting temperature rise versus airflow data clearly demonstrates the feasibility of 15W QSFP-DD modules in real-world data center environments.
QSFP-DD modules provide flexible, low-cost solutions that leverage vast experience in system, module, and cage thermal design and management strategies. Thermal testing confirms that the form factor also offers the greatest flexibility with regard to product customization. In both stacked cage and belly-to-belly configurations, QSFP-DD modules can support the thermal loads required to meet the demands for next-generation high-bandwidth applications.
By Scott Sommers, group product manager, Molex, and co-chair of the QSFP-DD MSA.
Please note: Important changes that will impact wattage levels are expected in mid-2019. Look for an update to this article later this year.
Like this article? Check out our other Connector Basics, High-Speed, Molex, and 2019 articles, and our Datacom/Telecom market page.
- QSFP-DD Strategies Help Data Centers Keep Cool - March 12, 2019
