Optical interconnects can meet data center challenges
Article By : Jeff Hockert
Data center networks can be revolutionized with optical I/O directly designed in server chips and packages.
Data centers are fast approaching an inflection point, where the performance gains from electrical interconnects grind to a halt, and the power consumption required to pump electrons through copper continue to escalate.
Fortunately, optical interconnects can meet the industry’s performance and power challenges, taking data centers into a new era of integrated photonics connections. At the recent Intel Labs Day, James Jaussi, senior principal engineer and director of the PHY Research Lab at Intel Labs, said “as we look to the future of communications and data center performance, there is a clear inflection point between electrical and optical.” We are quickly approaching the practical limits of electrical performance, such that without fundamental innovations, we see limits for energy-efficient circuit designs, he added.
The second challenge, Jaussi said, is the I/O power wall. “Bandwidth demand for compute approximately doubles every three years. Unfortunately, electrical performance scaling is not keeping pace with bandwidth demand—resulting in a power wall, where the I/O power is trending larger than the total available socket power—leaving nothing for compute.”
Electrical connections currently prevail over short distances, including board-to-board and package-to-package. That is set to change, as silicon photonics achieves key integration milestones, resulting in smaller optical modules, lower costs, and higher volumes.
Figure 1 Silicon photonics are finding use in data center environments. Source: Intel
Intel’s long-term performance goals are 1 Tbps of bandwidth per fiber at power consumption of 1 picoJoule per bit (pJ/b) with a range up to one kilometer. “By achieving these targets, we can potentially integrate 10s or 100s of Tb/s of bandwidth per socket, while fitting within reasonable power constraints, and simultaneously solve the reach limitations of electrical I/O,” Jaussi said.
In a traditional server system, high bandwidth electrical I/O is limited to less than 12 inches of internal routing distance due to frequency dependency losses. Scaling electrical connections between processors increasingly requires more I/O pins, more PCB layers and traces, and an increase in package size to meet the bandwidth demand – adding significant costs while doing nothing to solve the power wall.
With optical interconnects, “by tightly integrating optical I/O using silicon photonics, we can gain three benefits in a smaller footprint: lower power, higher bandwidth, and reduced pin count,” Jaussi said.
While much work remains to reach these goals, the path toward solving these challenges is clear.
Optical interconnect: Four breakthroughs
Over the past 15 years, Intel has integrated the laser on the photonics die. More recently, Intel researchers have described four additional technology breakthroughs that will bring integrated photonics into Intel-based server platforms, opening up new computing architectures that are more disaggregated. Multiple functional blocks such as compute, memory, and networking elements will be spread out and interconnected optically.
These four technologies include a micro-ring modulator architecture, an all-silicon photodetector, silicon optical amplifiers, and CMOS circuit integration. Haisheng Rong, principal engineer and manager of Silicon Photonics Device Research at Intel Labs, said that current modulator technology is relatively large-scale, measured in millimeters, while micro-rings are measured in microns—a >1000 times size reduction, which opens the door to placing hundreds of the devices on a server package. In 2020, Intel described direct modulation of micro-rings with CMOS circuits at >100 Gbps.
Figure 2 Intel’s optical network micro-rings writing wavelength division multiplexing (WDM) with micro-ring modulation. Source: Intel
In optical data transmission, the micro-ring modulator has the important role of encoding data onto the laser light. As light travels, it hits a virtual fork in the waveguide next to the micro-ring. Here the light can either continue or be trapped by the micro-ring modulator when a voltage is applied from the circuitry above it. When light is trapped in the micro-ring, it appears as a digital zero at the photo detector located at the end of the fiber cable. If the light continues, it appears as a digital one. The modulating voltage applied to the micro-ring creates the digital ones and zeros.
By using WDM lasers, separate micro-rings can modulate a different wavelength of light. “Using four micro-rings, we are able to trap four separate wavelengths from the same laser to convey four bits of data in the same beam of light,” Rong said. “With multi-wavelength micro-ring modulator technology, we can scale beyond four to eight, 16, and possibly more.”
Second, Intel has succeeded in developing an all-silicon photodetector. For decades, it was accepted as a fundamental limitation that silicon has virtually no light detection capability in the 1.3 – 1.6 μm wavelength range. That assumption was proved wrong in 2020 when Intel demonstrated light-detection capability with an all-silicon photodetector. A major advantage is processing and material cost reduction. Equally important, this silicon photodetector operates at a data rate of 112 Gbps.
Figure 3 Here Intel’s integrated photonics prototype highlights micro-ring modulators. Source: Intel
Third, as we focus on reducing total power consumption, silicon optical amplifiers are an indispensable technology. It is challenging to generate all the required light power from a laser while maximizing the energy efficiency. Fortunately, these optical amplifiers are 2-3 times more efficient, while made from the same material used for the integrated laser.
The ability to integrate the laser makes the amplifier possible. Moreover, the amplifiers work with WDM lasers, demonstrating impressive output power and energy efficiency.
Finally, “the co-integration of CMOS circuitry and silicon photonics brings all these technologies together,” said Ganesh Balamurugan, principal engineer at Intel Labs. Intel’s 100-Gbps silicon photonic transmitter consists of two stacked ICs: a silicon photonic IC on the bottom, which includes an integrated laser and the micro-ring modulators; above it is a CMOS electrical IC with the interface electronics. The two ICs are assembled in a 3D fashion using copper pillars, integrating the micro-rings and CMOS control circuitry in a single package.
“The electrical IC in our photonic transceiver houses all the electronics that are required to drive and control the ring modulator, for optical intensity modulation,” Balamurugan said. “Using high speed CMOS switching circuits, we can reliably transmit multi-level data by modulating the bias voltage of the ring modulators.”
Micro-ring modulators are extremely sensitive to temperature and process variations. To mitigate this, the electrical IC controls heaters integrated with the ring modulators to tune and stabilize their resonant wavelength. The control circuitry keeps the rings at a steady temperature and is essential to making the sensitive modulators commercially viable, Balamurugan said.
Optical transceiver prototype
Integrated photonics technology has the potential to be >4 times lower power than electrical I/O while simultaneously providing >1,000 times more reach.
Jaussi noted that some non-Intel technology platforms also include CMOS transistors on the photonics die. “Intel’s technology strategy is to keep our silicon photonics platform separate from our CMOS process technology,” he said. “This enables us to use the most advanced process technology, as it’s developed independently of the silicon photonics platform.”
The photonics and CMOS chips are integrated using techniques developed within Intel’s Assembly, Test and, Technology Development group. Co-integration is key to delivering performance- and cost-optimized optical transceivers.
Intel Labs has combined all the above critical technology ingredients in its latest transceiver prototype. The path is clear for integrated photonics to bring optical I/O into the server platform, integrated onto Intel server packages. Both the photonic and electronic ICs have been designed to support four-way WDM, quadrupling the data throughput compared to single-wavelength implementations.
The photonic IC includes a single cavity four-wavelength laser, on-chip optical amplifier and efficient ring-based modulation/detection with WDM. The CMOS electronic IC includes interface circuits to transmit, receive, and control all four wavelength channels. Intel has demonstrated error-free data transmission between transmit and receiver channels over on-chip silicon waveguides.
Jaussi concluded by noting that these technology advances, taken together, create a path to revolutionize the data center network by bringing optical I/O directly into our servers and onto our packages.
Performance varies by use, configuration, and other factors. Intel technologies may require enabled hardware, software, or service activation. Learn more at Intel’s Performance Index Overview.
This article was originally published on EDN.
Jeff Hockert is interconnect marketing manager at Intel Corp.
Editor’s note: This is the third article in a series about high-speed interconnect design.
Other articles in this series:
- High-speed interconnect requirements push for less expensive PCB materials
- CXL initiative tackles memory challenges in heterogeneous computing
Related articles:
- Silicon photonics: Will the hare finally catch the tortoise?
- A practical guide to signal integrity in high-speed SerDes applications
- The network-on-chip interconnect is the SoC