The Evolution, Proliferation, and Future of PCIe

By Martin Ng | December 08, 2020

PCI Express is set to release its Generation 6.0 specification in 2021, expanding speeds and bandwidth, and enabling new capabilities needed for cloud-based architectures. A new generation of PCIe connectors is ready now.

PCI Express (PCIe) is a popular peripheral bus architecture widely employed in enterprise servers, storage devices, cloud computing equipment, PCs, mobile devices, industrial computing and automation systems, and IoT and IIoT applications. Initially developed as a standard for display adapters (e.g., graphics cards and video cards) in desktop computers, this point-to-point, memory-oriented load/store protocol has since evolved into one of the most popular, if not virtually inevitable, solutions for mainstream computer-to-peripheral communications. In fact, PCIe has stood the test of time for nearly two decades now, spanning six generations of bandwidth expansion, and has continually outperformed customer expectations with regard to both high-speed performance and achieving economy of scale.

PCI Express

PCIe Evolution

In 2002, the PCI Express Generation 1.0 specification made it possible for a computing device to transfer data to a peripheral and back at speeds of 250MB/s each way per single lane of traffic.1 The current iteration, PCIe 5.0, can transfer data to and from peripherals at speeds of 4Gb/s per direction per lane — 16 times faster than its first predecessor — and in heavy-workload, high-bandwidth applications with 16 lanes (x16), can achieve collective unidirectional data transfer speeds of 64Gb/s or total speeds of 128Gb/s.1,3 The sixth-generation specification is currently under development and upon release, which is scheduled for 2021, will offer 8Gb/s per direction per lane, collective unidirectional speeds of 128Gb/s (x16), and total bandwidth of 256Gb/s.1,4

PCI Express bandwidth table

PCI-SIG

The PCI Special Interests Group (PCI-SIG) is comprised of more than 700 member companies committed to advancing its non-proprietary peripheral component interconnect (PCI) technology, including processor manufacturers such as AMD, Intel, NVIDIA, and Qualcomm, instrumentation specialists like Keysight and Teledyne, and original equipment manufacturers (OEMs) like HP and Dell. Each year, members collaboratively revise the specifications to further accelerate the development of high-speed devices capable of meeting ever-increasing demand for faster data transfer and ensure that each new generation is backward-compatible with the previous to better achieve economy of scale. The group’s commitment to these annual improvements has made PCI Express an especially popular choice for peripheral-to-computer communications.

PCI-SIG owns several standards designed to ensure operational compatibility with the PCIe protocol, including U.2, which was formerly known as SFF 8639; M.2, which was formerly known as the Next-Generation Form Factor or NGFF; OCuLink, a Thunderbolt competitor named for optical copper (Cu) links and designed for cabled PCIe connectivity; and Mini PCIe. Others, like the Compute Express Link (CXL), Cache Coherent Interconnect for Accelerators (CCIX), Non-Volatile Memory Express (NVMe), Gen-Z, and Enterprise and Data Center SSD Form Factor (EDSFF) standards are owned by partner groups.

PCIe Application Expansion

Since its introduction in 2002, PCIe has evolved into a leading data transfer protocol for computer-to-peripheral communications and has expanded its device support to include network adapters, add-in cards (AIC), network interface cards (NIC), acceleration cards, and NVMe flash drives in addition to the display adapters like graphics and video cards. This expansive and ongoing evolution is due to both the continued support of the PCI-SIG and four primary technology characteristics: low latency, memory coherence, high bandwidth, and economy of scale.

Low Latency — Networking in datacenters is considered to be dominion of Ethernet and InfiniBand, both of which are known for their high bandwidth. But when it comes to latency, these technologies can be far from ideal — especially in terms of intra-port data hops within a switch and inter-switch data hops. Designers who use PCIe-based application-specific integrated circuits (ASICs) for switching have achieved latencies down to just a hundredth of a typical Ethernet data hop. So, what PCIe lacks in speed (32Gb/s for PCIe 3.0 vs. 50–100Gb/s for Ethernet and InfiniBand), it makes up in reduced latency. PCI-SIG has also approved the use of Mini SAS HD connectors and cable assemblies, which can nearly match the performance of Ethernet-based interconnects like QSFP, for datacenter switching applications in which latency is more important than bandwidth.

Memory Coherence — Communication between host devices or processors and peripheral devices or workload accelerators like field-programmable gate arrays (FPGAs) has been subject to vendor constraints over shared memory. Modern FPGAs have in-built PCIe physical layers (PHYs) that facilitate cache-coherent connectivity or shared memory utilization. Since PCIe is a vendor-neutral standard, workload accelerators and processors can work in sync based on shared memory, which makes graphics processing unit (GPU) operations like machine learning faster and more effective. Two well-known accelerator technologies are CCIX, which uses PCIe 4.02, and CXL, which uses PCIe 5.0.

High Bandwidth — Embedded systems are getting smaller and more complex and are being built to process and transmit very large volumes of data, which is beginning to blur the line between these systems and the cloud enterprise. Single-board computers (SBCs) are increasingly expected to compete with the processing capabilities expected of high-end cloud-based systems like processors, GPUs, and FPGAs but within a comparative nutshell. PCIe-based SBCs can tap into that enhanced processing power with improved efficiency and are easier to plug into servers or associated memory than USB-based SBCs because they’re more processor-friendly. Similarly, in 360° embedded vision systems with high-definition (HD) cameras built around a small board, the combined throughput of the camera data, which has to be transported simultaneously through a cable to the central processing unit (CPU), can extend up to 32Gb/s. PCIe is an optimal solution for these applications as well due to its ability to deliver higher bandwidth and lower latency than existing protocols like low-voltage differential signaling (LVDS) and USB.

Economy of Scale — Artificial intelligence (AI) is being deployed for Big Data analysis in a number of increasingly diverse fields ranging from voter turn-out prediction to natural disaster management. The data behind these predictions is gathered from nodes of neural networks accelerated by digital signal processors (DSPs), FPGAs, and GPUs and trained on Big Data. PCIe plays a major role in transporting data from general server processors to high-performance chips and has helped churn out various data sets used to generate forecast models and make critical inferences and predictions. Major chip manufacturers like Intel and Qualcomm are seeing inference processing as an opportunity to expand on their AI capabilities. Inference processing is a less computing-intensive task than training; so, as Intel’s NNP-I and Qualcomm’s Cloud AI 100 solutions demonstrate, a lot of the corresponding PCIe-based hardware can easily be mounted on small-form-factor M.2 sticks and simply plugged into servers to achieve large-volume solutions with reduced cost of ownership.

PCIe Connectors

Connectors are an integral part of the PCIe ecosystem and must be designed according to strict PCIe specifications in terms of both performance (e.g., impedance, insertion loss, return loss, and crosstalk limits) and backward compatibility. Some PCIe connectors also have to be sturdy enough to withstand the weight of high-performance GPU cards and pass shock and vibration tests. In addition, all PCIe standards (whether owned by PCI-SIG or a partner group) are specified for use with specific connectors.

PCIe standards table

PCIe standards and their corresponding connectors.

Standard PCIe 1.0 to 6.0 connectors are defined by their lane count, which spans x1 to x16. For example, standard PCIe 4.0 and 5.0 connectors can transfer data at respective speeds up to 16Gb/s and 32Gb/s over eight channels or lanes.

SAS PCIe 4.0 and 5.0 connectors adhere to the U.2 (SFF-8639) and U.3 (SFF-TA-1001) specifications and are often used in data center storage flash arrays connected with enterprise solid-state drives (SSDs).

M.2 connectors deliver similar PCIe 4.0 performance in a smaller form factor and are ideal for both wireless connectivity applications in consumer electronics devices and AI processing in data centers.

Mini SAS HD connectors are available as overmolded external versions that connect to data center switches and as internal cable assemblies designed to latch onto high-speed data storage devices like enterprise SSDs. Another alternative for internal Mini SAS HD cable connectors is the PCIe OCuLink.

In addition, built-in proprietary systems like the Mini Cool Edge Connectors from Amphenol ICC offer multi-protocol support like EDSFF and Gen-Z over PCIe that makes them ideal for use in new EDSFF-based data center SSDs and FPGA-based workload accelerators.

Mini cool edge connectors from Amphenol ICC for PCIe

Mini Cool Edge Connectors from Amphenol ICC have a compact card-edge form factor with a high-density 0.6mm pitch and support high-speed performance over various PCIe protocols, including SFF TA-1002, Gen Z, EDSFF, and OCP NIC 3.0. They are available in vertical, right-angle, straddle-mount, and orthogonal configurations, transmit 32–56GT/s and up to 112GT/s PAM4, and are ideal for use in SSD, AIC, and NIC applications.

PCIe 6.0

PCIe 6.0, which is slated for release in 2021 and expected to have hardware available by 2023, packs in more features than ever before.1 It doubles PCIe 5.0 bandwidth capabilities by using pulse amplitude modulation (PAM4) signaling rather than non-return to zero (NRZ) signaling and, since PAM4 is more error-prone than NRZ, incorporates forward error correction (FEC) as well, which adds design complexity but achieves higher speeds: 8Gb/s per direction per lane, collective unidirectional speeds of 128Gb/s (x16), and total bandwidth of 256Gb/s.1,4 The PCIe 6.0 rollout may be costlier to implement than 5.0 but should also prove more beneficial to cloud-based non-volatile storage and AI accelerators than to consumer electronics like gaming applications, which will further enhance its value.

PCIe 6.0 Specification

This graph from PCI-SIG illustrates how PCIe I/O bandwidth rates have doubled every three years.4

The Road Ahead 

Thanks to solid industry-wide support and the dedication of PCI-SIG and various partner groups, PCIe has consistently demonstrated its ability to remain relevant, if not ahead of its time, for nearly 20 years and counting.

For more information, visit Amphenol ICC online.

References:

  1. Smith, Ryan. “PCI Express Bandwidth to be Doubled Again: PCIe 6.0 Announced, Spec to Land in 2021.” AnandTech. June 18, 2019.
  2. Yanes, Al. “PCI-SIG DevCon 2017 Update.” PCI-SIG. 2017.
  3. Yanes, Al. “PCI-SIG DevCon 2018 Update.” PCI-SIG. 2018.
  4. Yanes, Al. “PCI-SIG DevCon 2019 Update.” PCI-SIG. 2019.

Like this article? Check out our other standards, cloud computing, and new technology articles, our 2020 and 2019 Article Archives, and our Datacom/Telecom and Industrial Market Pages.

Martin Ng
Latest posts by Martin Ng (see all)
Get the Latest News
x