PCI Express and ASI: Avoiding chaos at 2Gbps

PCIexpress (PCIe) is rapidly becoming the dominant standard for high-speed data transmission in PCs, notwithstanding the existence of myriad other high-speed point-topoint serial protocols, most of which predate PCIe. What’s more, elements of PCIe, particularly the physical layer, may become even more pervasive as part of ASI (Advanced Switching Interconnect), an ambitious protocol targeting applications that require performance higher than that of typical desktop and laptop PCs. The ASI SIG (Special Interest Group) names communications, storage, servers, and embedded systems as target markets but does not limit ASI’s potential to those markets.

ASI appears to bring many benefits to high-speed datatransmission. For example, unlike the host-centric PCIe, ASI enables peer-to-peer communication among a system’s peripheral devices—or among CPUs in systems that incorporate multiple CPUs. Because it allows multiple conversations to take place simultaneously among these devices, the peer-to-peer feature can greatly speed information flow in complex systems. Other ASI features that PCIe doesn’t support include QOS (quality of service); protocol tunneling, which enables ASI-based systems to support multiple protocols; and the fabric switching that is at the heart of ASI.



Nevertheless, according to its detractors, ASI faces stiff competition from older protocols, such as RIO (RapidIO)—especially with the new RapidFabric extensions, InfiniBand, Fibre- Channel, and a sizable list of proprietary protocols that have established substantial bases of users, most of whom allegedly see no particular advantage in changing horses for the foreseeable future. Many in the industry regard Ethernet as a key competitor, but the ASI SIG’s position is that that long-lived protocol complements and does not compete with ASI.

Current single-lane PCIe implementations transmit data at a maximum rate of 2Gbps in each direction. The raw data rate is 25% higher—2.5Gbps—because PCIe, like most high-speed-serial protocols, uses 8B/10B (8-bit/10- bit) coding to embed the clock within the data stream. However, both PCIe and ASI allow the use of multiple lanes, which can increase data-transfer rates by factors as large as 32. Already, advanced video subsystems in some premiumpriced desktop PCs that target video gaming are using unidirectional 16-lane-wide versions of PCIe to transmit data at 32Gbps—equivalent to 4Gbytes/ sec. Despite such mind-numbing speed, these desktop systems don’t require ASI’s advanced features, so a unidirectional version of PCIe is the appropriate choice.

Rapid deployment
Manufacturers of PCs and peripherals are deploying PCIe at a rate that, figuratively speaking, rivals the rates at which data travels on the bus (Reference 1). And these manufacturers seem undeterred by difficulties in obtaining from the PCI SIG certain detailed information that could come in handy to engineers creating new equipment designs. A key reason that gaps in the information that the SIG has so far provided have not hindered adoption of PCIe is the voluminous information available from the PICMG (PCI Industrial Computer Manufacturers’ Group). PICMG’s role complements the roles of protocol-specific industry groups, such as the PCI SIG and the ASI SIG. Although PICMG defines no communication protocols, it does publish system-physicalarchitecture specifications that enable designers to successfully implement protocols that other groups have defined. Moreover, PICMG member companies provide the hardware for these implementations. Recently, PICMG completed what it calls the largest spec-writing effort in its 11-year history. The result is a group of specifications collectively describing the modular ATCA (Advanced Telecommunications Computing Architecture).



Among many others, the PCI SIG’s PCIe specs and the ASI SIG’s ASI specs complement the ATCA specs-although PICMG prefers to call the protocol specs subsidiary to ATCA. Versions of the ATCA spec define how to implement the architecture with different communication protocols. One spec covers both the ASI and the PCIe versions. This commonality should come as no surprise because the two protocols share both their physical and data-link layers, although ASI’s switch fabric will almost certainly require module-resident hardware that PCIe doesn’t use. What may come as a surprise, though, is that PICMG has so far found that one card-cage design, one backplane design, and one module format are sufficient to implement versions of ATCA that support at least half a dozen communication protocols.

Module manufacturers have even developed prototype plug-in ATCA-format line-card modules that they can adapt to any of several communication protocols simply by loading the appropriate software. However, according to Gilles Garcia, director of marketing for switched-fabric products at AMCC (Applied MicroCircuits Corp), line cards built around network processors need more than software to adapt to ASI or RIO/RapidFabric. These protocols require additional glue logic implemented in FPGAs. Garcia sees no motivation to develop new network processors that subsume this external logic because the marketplace isn’t yet demanding line cards that support ASI or RapidFabric.

Promising implications
Nevertheless, ATCA’s implications appear promising indeed: In telecom central-office equipment, the architecture appears poised to become a unifying force that will enable competing companies to produce
physically similar hardware. This unification will bring about component-level economies of scale that help to control costs and drive product differentiation to the system-software level, at which many believe it belongs. Indeed, ATCA visionaries see the architecture’s transcending the telecom market space and invading such computing areas as clustered servers, producing further economies of scale.

While it was busily at work on ATCA, PICMG scarcely ignored less complex systems. Always mindful of its roots in the older, parallel versions of PCI, the organization has focused on updating architectures that it pioneered for smaller systems. The best known of these is CompactPCI. PICMG is nearly ready to announce a specification for a PCIe-based version of this architecture. The group expects the old and new CompactPCI architectures to coexist for many years as the PCIebased architecture gradually supplants the parallel version in new equipment designs. PICMG is also specifying a 95_125mm PCIebased mezzanine-card format that not only can supplant earlier mezzanine cards, but also may replace PC-104 boards in some embedded-system applications.

Many people think of PCIe as merely a bus for interconnects on and between pc boards within a system enclosure. Also, despite the multilane configuration that highend video subsystems use, most applications use a single-lane, bidirectional configuration comprising two signal pairs.

Still, needs exist to run PCIe through cables and to place multilane versions in backplanes. For the backplane application, which will be common in ASI-based systems, designers have a choice of connectors that are mechanically and electrically acceptable.

Prices vary, but some of these parts deserve to be called downright inexpensive when you consider their amazing electrical performance. At press time, however, the PCI SIG had not named one design as the “official” PCIe backplane connector, although PICMG’s ATCA specs call for the ZD series, currently available from Tyco/Amp and ERNI. Though ATCA does not reference them, other backplane connectors, including Teradyne Connection Systems’ GbX series, are electrically suitable.

And, although rumors exist that the PCI SIG has made an internal choice of a cable connector, no official announcement has emerged, and the vendor whose design the SIG is rumored to have selected refuses to make any comment—or even to provide information about the design. Purportedly, the need to resolve licensing issues has caused the announcement delay.



Nevertheless, Rajeev Kumar of Intel, chairman of the ASI SIG, says that he has observed good results with routing of PCIe signals over cables as long as 7m that were designed for other standards. He mentions Molex cables that were designed for SAS (serial-attached SCSI) and Gore CX4 cables that were designed for 10Gbps Ethernet. He adds that, in some cases, you can obtain electrically and mechanically equivalent cables from several sources.

A bit easier than you may think
Just the idea of transmitting data error-free at 2Gbps and more can be intimidating. Nevertheless, the task, though far from trivial and not to be approached lightly, may be easier than many people suspect. Not too many years ago, the only engineers who would even consider dealing with such high frequencies were microwave engineers. However, whereas the frequencies in PCIe data streams and microwave circuits are comparable, the technologies have less in common than you might think.

The reason is that the serial buses carry information in digital form, whereas classical microwave designs are traditionally analog. Despite the millivolt signal levels in the differential pairs that carry information on buses such as PCIe, you generally need to know the voltage levels with only about 20% accuracy; the voltage differences between the one and the zero levels are usually greater than the absolute voltage on either line of the differential pair. In other words, the buses work because they were designed to tolerate errors.

As a result, a reasonable amount of ringing after a one-to-zero or zero-to-one transition is not usually fatal. Still, discontinuities and reflections in transmission paths can and do cause errors, so you must design pc boards, connectors, backplanes, and cables to minimize crosstalk and to maintain a constantimpedance environment with minimal discontinuities, and you must also minimize delay differences that cause skew between signals that are supposed to reach receivers simultaneously.

In addition, when you work with signals at several gigahertz, the dielectric losses in pc substrates and cables become appreciable, attenuating signal swings and slowing (lowpass-filtering) signal edges. In all probability, without the introduction of pre-emphasis into the circuits that drive high-speed serial buses, the design problem would be orders of magnitude more complicated, and data reliability that designers now take for granted would be unachievable.

Adaptive equalization
The object of pre-emphasis is to shape the edges of signals that leave the driver so that the distortion (lowpass-filtering) introduced by the dielectric losses produces nearly ideal waveforms at the receiver input. But, because the amount of distortion depends on material properties and the length of signal paths, the tuning problem could be intractable were it not for adaptive equalization. You can find adaptive equalization in some IP (intellectual-property) cores for ultrahigh-speed serial buses. At 2.5Gbps, the current generation of PCIe is too slow to require the technique, but if the data rate reaches 5Gbps, future generations may.

Adaptive equalization simplifies the task of designing pc boards, backplanes, and cables that work reliably at multigigabit-per-second data rates. And it can do so without requiring tedious manual tuning or the use of pc substrates and cable dielectrics made of low-loss materials whose costs greatly exceed those of familiar materials. However, even with the use of adaptive equalization, ultrahigh-speed designs still require extreme care. For example, unless the componentlevel circuit models accurately describe the component behavior, no simulation of a multigigahertz design can accurately predict the performance of the complete circuit. In the area of how best to characterize and model components such as connectors, backplanes, and cables, the industry’s signal-integrity experts seem to be embroiled in a controversy (see box “Don’t let fixture performance invalidate connector measurements”).

Don’t let fixture performance invalidate connector measurements by Orlando Bell, Giga Test Labs The PCI SIG (Special Interest Group, www.pcisig. com) has published the “PCI Express- Connector High-Speed Electrical-Test Procedure.” This document describes the methodology for measuring the performance of PCI Express connectors to 10GHz, using a custom test fixture and a four-port VNA (vector-network analyzer). You can also use a more accurate and flexible probe-based methodology for such high-speed serialinterconnect measurements. Fixture-based validation of connector designs is desirable for several reasons. First, the drive for industry standards works best when all interested parties avoid correlation problems by using consistent test hardware. In addition, using SMA connectors simplifies implementation of the fixturing interface ( Figure A ) because SMAs are the de facto standard connections for highfrequency measurements. Unfortunately, in many cases, connectors and traces on high-speed test fixtures can make the fixtures’ performance inferior to that of the devices the fixtures are supposed to help characterize. Often, a fixture exhibits signal loss greater than that of the device under test. If the device is inherently a two-port element-that is, a device that has one input and one outputwith little crosstalk between the ports, you can use T-matrix manipulation to subtract the fixture contribution. On the other hand, if the device under test has four ports that suffer from fixture-level crosstalk, de-embedding the fixture contribution becomes difficult. To overcome these fixture limitations, you must usually move the measurement reference plane as close to the device under test as possible. You can achieve this close spacing by using microwave probes ( Figure B ). By using a calibration substrate ( Figure C ), you can de-embed the measurement system’s response all the way to the probe tips. In many cases, you can directly probe the device; in the worst case, you can add vias to the board for use as probing points. Either way, the measured data usually requires little or no de-embedding. When pins on which you make no measurements require termination, you encounter the sole disadvantage of using probes. Then, you must use SMT (surfacemount- technology) resistors to terminate the pins. The red plots in Figure D show a connector’s TDR (time-domain-reflectometer) performance as measured with a fixture that has SMA connectors plus a 6-in. stripline trace on an FR4 substrate. The blue plots show what happens when you make the measurements with a microwave probe. The 10 to 90% rise time of the TDR step, is 200psec in Figure Da and 500psec in Figure Db . You can readily appreciate the fixture’s effect on the connector’s response: The fixture degrades the measured impedance and delay. Also, extracting an equivalent-circuit model of the fixtured connector requires extensive effort, because you must include the fixture in the response. Doing so adversely affects the model’s quality and bandwidth. The use of fixtures is desirable to ensure compatibility and correlation, but you pay for this convenience with reduced measurement accuracy and reduced bandwidth in the device you are testing. Author’s Biography Orlando Bell is vice president of engineering at GigaTest Labs in Sunnyvale , CA . His key responsibilities include managing the highfrequency- characterization lab and driving the research and development of advanced test hardware for the signal-integrity market. He holds bachelor’s and master’s degrees in electrical engineering from the University of Florida ( Gainesville ).

To quickly and economically get new designs to market, designers need rules that they can apply easily and without a lot of thought. To this end, the PCI SIG and its member companies have gone to unprecedented lengths and great expense to provide component characterization and design-rule information that engineers must consider in their designs.

Nevertheless, some signal integrity consultants, such as Edward P Sayre, PhD, president of NESA (North East Systems Associates Inc), say they are concerned that a few designers may interpret the SIG’s approach as, “We’ve done all the thinking and encapsulated the results in rules; all designers need to do is follow the rules without thinking.” Says Sayre, “That’s not what the SIG intends, but if that’s what some engineers believe, the ones who haven’t given their designs adequate consideration are going to wish that they had taken a little more time to understand why their circuits and layouts behave as they do.” Sayre describes the SIG’s work as impressive, but he says that years in the field have taught him that, although design rules are invaluable, they can’t replace an understanding of the underlying circuit behavior. Moreover, the most troublesome problems seem to occur when designers fervently, albeit erroneously, believe that they have followed all of the rules but haven’t developed an understanding of the rationale behind those rules.

Author information
Contributing technical editor Dan Strassberg, who has covered test and measurement for EDN since 1987, occasionally ventures into other product areas. For example, this article started out as a passive-components story. However, the larger issues of how to properly think about high-speed serial buses soon grabbed the spotlight. Strassberg holds a bachelor’s degree in electrical engineering from Rensselaer Polytechnic Institute (Troy, NY) and a master’s in electrical engineering from the Massachusetts Institute of Technology (Cambridge).

Reference
1. Bhatt, Ajay V, “Creating a PCI Express Interconnect,” www.pcisig.com/specifications/pciexpress/resources/PCI_Express_White_Paper.pdf.

For more information... For more information on products such as those discussed in this article, contact any of the following manufacturers directly, and please let them know you read about their products in EDN Asia. Trade associations and industry groups Suppliers of ICs and intellectual property for embedding in ICs AMCC (Applied MicroCircuits Corp) Suppliers of connectors, cables,and related components Suppliers of test equipment, development tools, or consulting services ASI SIG (Advanced Switching Interconnect Special-Interest Group) www.asi-sig.org AMCC (Applied MicroCircuits Corp) www.amcc.com ERNI Electronics Inc www.erni.com Agilent Technologies www.agilent.com ATCA (Advanced Telecom Computing Architecture) www.advancedtca.org Intel Corp www.intel.com FCI www.fciconnect.com CATC (Computer-Access Technology Corp-subsidiary of LeCroy Corp) www.catc.com InfiniBand Trade Association www.infinibandta.org StarGen Inc www.stargen.com ITT Cannon www.ittcannon.com GigaTest Labs www.gigatest.com PICMG (PCI Industrial Computer Manufacturers’ Group) www.pcisig.com Synopsys Inc www.synopsys.com Molex www.molex.com NESA (North East Systems Associates) www.nesa.com RapidIO Trade Association www.rapidio.org Xilinx Inc www.xilinx.com Tyco/AMP www.amp.com TDA Systems Inc www.tdasystems.com SCSITA (Small Computer Systems Interface Trade Association) www.scsita.org   WL Gore and Associates www.gore.com Tektronix Inc www.tektronix.com VITA (VMEBus International Trade Associations) www.vita.com