Largely unknown to users, the PC industry, with embedded computing following close behind, is in the middle of a quiet shift in technology. The overtaxed PCI (Peripheral Component Interconnect) bus for add-in cards is slowly giving way to the serial PCIe (PCI Express). This change is bringing new capabilities and performance levels to desktop-, laptop-, and embedded-computing systems, but it is also creating short-term design challenges with the technology evolving faster than its adoption.
Now nearly 15 years old, the PCI bus has stretched to its limits. As processor-clock speeds and I/O demands have increased over time, the PCI bus evolved higher speeds and wider buses to boost system performance. But, as with all other parallel-bus structures, problems with the increase in skew and fan-out have also grown with each step in PCI’s evolution. The bus years ago reached its limits, prompting the PCISIG (PCI Special Interest Group) to develop a next-generation version that could support legacy applications and software. The PCISIG’s answer was PCIe, a switched-serial-bus structure with lower layer hardware that hid this fundamental architectural change from upper layer software.
Links in PCIe are composed of lanes that carry high-speed, bidirectional, serial data point to point within the system. These links have scalable bandwidth and can use one, two, four, eight, 12, or 16 lanes to meet a peripheral’s performance needs. The hardware in the PCIe link handles the conversions between parallel and serial forms, the clocking and synchronization of multiple lanes, error detection and correction, and a variety of other tasks that might otherwise require changes in driver, application, and operating-system software. As a result, designers can make a PCIe-bus implementation virtually invisible to the user.
Because of this invisibility, the PC industry has been quietly switching its board and silicon designs to PCIe without consumers noticing the shift. A look at the PCISIG integrators list for February 2007 gives an indication of how the adoption of PCIe has progressed (Reference 1). Nearly a third of the components and silicon IP (intellectual property) on the list address endpoint-controller needs. Another third are graphics devices with native PCIe interfaces from companies such as AMD, ATI Technologies, and Nvidia. Most of the remainder are host- or I/O-root complex switches or bridges to the older PCI and PCI-X buses from companies such as Intel, NEC, PLX Technology, and Via Technologies. Only a handful are peripheral functions, such as Serial ATA or Ethernet controllers with native PCIe interfaces. In the board market, the PCIe integrators list shows nearly 100 graphics-board offerings, whereas other functions each have fewer than a half-dozen instances.
Graphics dominates
This pattern of offerings shows that, for the PC at least, the application driving PCIe adoption has been high-performance graphics. Other high-bandwidth-PC applications, such as Ethernet, Fibre Channel, InfiniBand, and Serial ATA, have also embraced PCIe, but much less enthusiastically. This pattern suggests that the PCI and PCIe buses will continue for many years to be present together in PCs to handle peripheral functions that have been slow to adopt PCIe. A quick look at motherboard offerings available from a local electronics superstore, for instance, shows that almost all PC motherboards have at least one PCIe card slot but still offer several PCI slots.
These surveys of PC offerings, however, do not tell the whole story. The PCMCIA (Personal Computer Memory Card International Association), for example, has also adopted PCIe as an interface for its ExpressCard standard, which it expects to replace CardBus for laptop add-in cards. This adoption extends the applicability of PCIe beyond what the PCISIG has defined, and it is not the only such extension. Among the PCIe silicon-chip offerings is silicon IP from companies such as Cadence, Mentor, and Synopsys for custom chip designs and IP for FPGAs from companies such as Altera, Lattice Semiconductor, and Xilinx. The availability of this IP, especially the cores for FPGAs, indicates a significant market for custom devices and boards with PCIe interfaces, such as the data-acquisition boards from Adlink and National Instruments. Often, these custom designs target the embedded-computing market.
PCs’ low cost, wide availability, high performance, and immense support infrastructure have long tempted developers to adapt PC standards and technology to embedded-computing applications. When the PCI bus became available, for example, the embedded-system community created CompactPCI, PXI (PCI Extensions for Instrumentation), and the SHB (system-host-board) passive-backplane embedded-computing architectures, among others, based on PCI technology. A new trade organization even arose: the PICMG (PCI Industrial Computer Manufacturers Group).
The same kind of thing has happened with PCIe. Predictably, CompactPCI Express, PXI Express, and SHBe have all adopted the PCIe-bus standard. In addition, new architectures that can use PCIe have arisen. In the demanding telecommunications market, for instance, PICMG has defined the backplane of its ATCA (Advanced Telecommunications Computing Architecture) and its AMC (Advanced Mezzanine Card) modules so that they can use PCIe as an interface. All in all, more than 60 board form factors and a variety of connector configurations with PCIe-bus interfaces are available for industrial and embedded computing (see figure).
Rapid evolution
However, the PCIe specification is evolving faster than designers are implementing. The original PCIe 1.0 specification appeared in 2002, with Version 1.0a arriving a year later and Version 1.1 arriving in 2005. Because the design time for new silicon is typically 18 months or more, many available chip implementations still follow the 1.0a specification, whereas only a few follow 1.1. Fortunately, there are only small differences between the revisions, so they are compatible.
In the meantime, however, the PCISIG has continued to evolve the PCIe specification. Revision 2.0 became available in January 2007, and it offers several significant changes to the specification. One of the most dramatic is an increase in the serial-data rate. The original PCIe specification called for the lanes to communicate at 2.5 Gbps in each direction; PCIe 2.0 doubles that rate to 5 Gbps. To maintain interoperability between the 1.1 and the 2.0 interfaces, the specification has made the speed increase optional and includes an LTSSM (link-training-and-status-state-machine) rate-negotiation feature to identify which speed both ends of a link will support.
Speed increase is not the only new feature of PCIe 2.0, however. In addition to incorporating a number of errata that its developers discovered during the design of Version 1.0a and Version 1.1 devices, the 2.0 specification has added new functions. One new function is optional programmability and a mandatory disable mechanism for signaling timeouts. This feature helps developers implement designs that may produce longer signal delays than they expect in a motherboard environment. Another optional capability is function-level reset commands, allowing system software to individually reset board functions instead of simply resetting the entire board. A third addition is the use of ACS (access-control service) to support peer-to-peer communications across the PCIe bus (see figure).
The PCISIG is not stopping there, however. In February 2007 it released a standard for extending the PCIe bus across a cable (see box “Multichassis PCI Express”). It is also working on specifications to extend graphics-card power levels to 300W, creating new form factors for add-in cards, and methods to provide I/O virtualization. This virtualization effort is a two-step process to allow a PCIe system to simultaneously behave as several machines using shared hardware. The first step is to define a single-root platform, which allows several operating systems and applications to run on a single hardware platform while appearing as multiple machines to a network. The second step is to define a multiroot structure that allows multiple processors to share peripheral hardware.
PCIe beyond the PC
All these innovations and enhancements to the PCIe specification are greatly expanding its applicability within and beyond the PC market. They are also creating some design concerns that, although they will eventually reach resolution, are becoming challenges for developers trying to apply the technology today. One of the biggest challenges today involves the boot firmware or BIOS, which starts the computing platform after you apply power.
According to Trevor Western, vice president of BIOS engineering at firmware vendor Phoenix Technologies, the challenge centers on how the BIOS should support operating systems and devices that do not offer PCIe or that incorrectly implement it. The backward compatibility to PCI that PCIe offers allows the BIOS developers to at least configure the system as if everything inside were PCI, but this approach is conservative. PCIe has capabilities, such as isochronous transfers, that the standard does not tightly define and that may have variations in implementation. Accessing those capabilities is risky, according to Western. An error in the implementation could lock the system during boot-up.
Western also notes that popular operating systems, such as Windows XP and Windows 2000 server software, do not explicitly support the extra features of PCIe, such as hot-plug capability, power management, and MSIs (message-service interrupts). As a result, the BIOS vendors generally disable PCIe functions in computers using these programs unless customers request such support. Some BIOS versions also allow you to turn on the PCIe features during system configuration. Still, developers should be aware of what OS their system uses to avoid disappointment when crafting their peripherals or embedded systems. If the OS or BIOS does not support MSIs, for example, the system reverts to PCI’s IRQ (interrupt-request) format and may be unable to attain full PCIe performance.
Western points out, however, that this situation is temporary and applies only to older motherboards, BIOS versions, and the computers using this older software. Windows Vista and the forthcoming Longhorn server software do support PCIe’s extra capabilities, as will the BIOS of systems designed for them. He also notes that proprietary embedded operating systems and Linux-based systems have had success with PCIe support.
BIOS poses challenge
The embedded-computing industry has found other BIOS challenges for PCIe developers, however. One is a problem with enumeration—the allocation of address, memory, and interrupt resources to peripherals during boot-up. Because PCIe is a switched architecture, the BIOS must look through the switch device to locate and identify the peripherals on the other side. Depending on the implementation, this process may cause the BIOS to see more peripherals requesting resources than it can handle, especially if multiple switches are in a cascade. As a result, the system may not enumerate properly.
Another challenge with switch use relates to spread-spectrum clocking. To reduce EMI, the PCIe physical layer can operate with a clock that varies slightly around the nominal 2.5-GHz rate. This variation spreads the energy spectrum of the clock edges into a band around 2.5 GHz, instead of concentrating them at the clock frequency. Spreading the spectrum in this way reduces the peak energy level at any one frequency, keeping system EMI within regulatory limits. Unfortunately, a PCIe switch may not support the use of this technique through its ports, so BIOS vendors typically disable this feature.
The embedded-system community has also become wary of using motherboards designed for desktop or server use. Developers report that some motherboards may have card slots with connectors that handle eight- or 16-lane cards but may route only four lanes for those slots. Having the larger connectors maximizes the possibilities for inserting boards into the system, but failure to route all the lanes causes the motherboard to get less than top performance from high-performance boards.
System designers will continue to face such challenges for the next several years. Standards developers are only now resolving the issues that arose with the adoption of PCIe 1.1, for instance. With PCIe 2.0 having just arrived, no silicon or software is yet available that will give developers access to its features, and experience has not yet revealed the inevitable ambiguities and errata that plague all new specifications. Further, there will be a transition during which both types of PCIe interfaces must work together as vendors migrate their products from one version to the next and customers slowly embrace them.
Still, developers will inevitably face and resolve those challenges. The performance and features available using the PCIe bus are compelling enough to ensure that desktop-, server-, and embedded-computing systems will quickly adopt PCIe. PCIe has been quietly slipping into nearly every type of computing system and will eventually supplant PCI. It just may take a while for it to all settle.
For more information
· Actel: www.actel.com
· Adlink: www.adlinktech.com
· Altera: www.altera.com
· AMD: www.amd.com
· ATI Technologies: www.ati.com
· Avnet: www.avnet.com
· Cadence: www.cadence.com
· Dalsa Inc: www.dalsa.com
· Intel Corp: www.intel.com
· Intel Developer Network for PCI Express: www.pciexpressdevnet.org
· Lattice Semiconductor: www.lattice.com
· Mentor: www.mentor.com
· Molex: www.molex.com
· National Instruments: www.ni.com
· National Semiconductor: www.national.com
· NEC: www.nec.com
· Nvidia: www.nvidia.com
· One Stop Systems: www.onestopsystems.com
· PCI Industrial Computer Manufacturers Group: www.picmg.org
· PCI Special Interest Group: www.pcisig.org
· PCMCIA Forum: www.pcmcia.org
· Pericom Semiconductor: www.pericom.com
· Phoenix Technologies: www.phoenix.com
· PLX Technology: www.plxtech.com
· Synopsys: www.synopsys.com
· Via Technologies: www.via.com.tw
· Xilinx: www.xilinx.com
Reference
· PCI Express Integrators List.
Multichassis PCI Express
One of the new functions that the PCISIG (Peripheral Component Interconnect Special Interest Group) has added to PCIe (PCI Express) is an ability to extend the bus outside the box. The PCIe External Cabling Specification, which PCISIG announced in February 2007, standardizes the connectors, pinouts, and cabling for bus expansion. It also imposes signal-level and noise-budget standards to ensure interoperability among vendor implementations. A number of companies, such as One Stop Systems, have taken advantage of this capability to create expansion systems that connect to the host PC through the PCIe cable. Experience shows that a simple cable can extend the PCIe bus as far as 7m, whereas a cable with active noise reduction can achieve 10m or more. Avnet, for instance, claims that the cable-extender evaluation kit that it created for Xilinx can achieve 15m distances when using signal-conditioning chips from National Semiconductor.
The serial nature of PCIe has a handy side effect when it comes to bus expansion: There is no need for conversion. Computer-peripheral buses have typically resembled variations on the processor’s external addressing and data buses, retaining their parallel nature. Because of skew and signal-integrity problems and the high cost of a cable containing dozens of wires, the parallel-bus structure could not readily extend more than 1m or so outside the chassis. To expand the bus any farther required conversion between the parallel bus and another format more suitable for transport. This scenario differs for PCIe. The bus is serial in nature and suitable for long-distance transport because it is self-clocking and automatically handles skew when it uses more than one lane. Because it does not require conversion, PCIe-cable expansion avoids added latency and can offer full system performance across the expansion.
A PCIe cable is not simply a mirror image of the PCIe bus, however. The cable does contain the differential PCIe-signal pairs, but it also incorporates additional sideband signals. The sideband signals include a differential, 100-MHz reference clock, as well as single-ended reset, board-present, and power-on signals. They also include single-ended signal return and several grounds. The cable may contain other optional signals, such as 3.3V power and power return for active connectors that offer noise reduction to extend achievable cable length, and a wake-up signal. Reserved signals allow for future expansion. This assortment of PCIe, sideband, and reserved signals—18 conductors for one lane to 136 conductors for 16 lanes—means that system designers must consider the impact of cable size when selecting an expansion-bus speed.
Cable opens new options
The expansion of PCIe across a cable allows development of systems and architectures that parallel-system-bus structures cannot achieve. System I/O is one function that can change dramatically through the use of PCIe cabling. By using the One Stop Systems MaxExpress four-lane PCIe-expansion products, for example, developers can create multichassis systems with as many as 100 add-in boards under the control of a single PCIe host.
Another application of PCIe cabling is bus and form-factor conversion. You can use the cable, for instance, to connect a laptop computer through an ExpressCard port to a CompactPCI chassis with a bus-bridge card. The laptop can then serve as the system host controller, effectively making the laptop into a CompactPCI system. Similarly, CompactPCI Express systems can incorporate passive-backplane extensions, and PCI-X boards can link through laptops. These scenarios require only a PCIe-cable interface for each type of chassis, and you can link a variety of form factors and bus types to form a single system.
A third possibility is creation of a host-to-host link over a PCIe cable. By using a nontransparent switch at one endpoint, two computing systems can link their system buses for high-speed, low-latency data transport across the PCIe cable. Systems can achieve data rates as high as 32 Gbps with 16-lane cables and latency equivalent to one system’s acting as an add-in card for the other.
These applications are only the beginning for PCIe cabling. Work within the PCISIG to define I/O virtualization and advanced switching will bring networklike features with all their advantages to PCIe. You will be able to connect PCIe systems to form powerful networks without the performance limitations of protocol conversion. Further, these systems can be mixtures of platforms and form factors with only one feature in common: a PCIe-cable port.
At a glance
· PCIe (Peripheral Component Interconnect Express) is quietly entering both the PC and embedded-computing markets, lifting the bus-bandwidth limitations of PCI without compromising legacy software.
· Enhancements to the PCIe specification are occurring faster than implementation, creating temporary challenges for system designers.
· Developers must carefully look at motherboard and BIOS selections to ensure that they can achieve maximum performance.
· PCIe cabling offers many new system-expansion opportunities.
Author Information
Contributing Technical Editor Richard A Quinnell has been covering technology for more than 15 years after an equally long career as an embedded-system-design engineer.
Click here for Illustrations:
Figure 1Figure 2
