Bandwidth needs continue to grow very rapidly at the top cloud data centers because of drivers such as video, AI, HPC and 5G. As these customers drive towards higher performance/scale solutions with a huge focus on carbon neutral/negative goals, they are looking for significantly higher power-efficient networking solutions through various initiatives. One key initiative to reduce power consumption in high-end networking systems over the next few years is Co-Packaged Optics.
What is Co-packaged Optics (CPO)
Today, the most popular 32x400G (12.8Tbps) switch solution consists of a 1 rack-unit (1RU) switch system, which has 32 cages (QSFP-DD or OSFP) on the front panel where the optics modules plug in. The optics modules connect to the switch silicon on the system’s PCB (printed circuit board), using high-speed SerDes I/O traces. Typically, longer reach SerDes connections consume higher power than SerDes designed for much smaller reach.
Figure 1: Today’s switch with pluggable optics modules
Co-packaged optics brings optics much closer to switch silicon die in a single package, so that power can be saved by reducing the reach of the SerDes. As shown in the figure below, a switch die on a package substrate, with optics tiles around it is called a co-packaged optics assembly (CA). In a CPO switch, the co-packaged optics assembly sits on a smaller PCB inside the switch chassis and there are no cages on the front panel for pluggable optics modules. Optical fibers come out directly from the optics tiles in co-packaged optics assembly and connect to fiber connectors such as MPO (multi-fiber push on) on the switch faceplate.
Figure 2: Co-packaged optics assembly
Figure below provides an architectural view of the co-packaged optics assembly (CA). It consists of a switch assembly (SA) and a number of optics assemblies (OA) which are also known as optics tiles/chiplets. These optics tiles/chiplets have optical components, similar to those found in pluggable optics modules.
Figure 3: Architecture of Co-packaged optics assembly
Delivering Power-Efficiency with Co-packaged Optics (CPO)
Bringing optics closer to switch silicon on a package substrate with a co-packaged optics (CPO) solution helps achieve the goal of power-efficient networking, as shown in the figure below. Rather than optics modules getting plugged into the front-panel cages, connected to the switch silicon using traces on a PCB with SerDes, optics tiles/chiplets would be co-packaged with switch silicon die so that power consumed by SerDes can be reduced significantly. Overall solution power could go down by 30% or higher.
Figure 4: CPO reduces power as optics is brought closer to switch silicon
Not only does co-packaged solutions help lower power, it also drives higher switch rack space density, as the front panel is not limited by cage space. Further, it can also provide higher uptime from fewer operator errors while plugging optics modules into the switch front-panel cages. It also promises to deliver better total cost of ownership (TCO).
Open, Disaggregated and Standards-based Solutions are Critical
Most successful IT products and solutions are open, disaggregated and standards-based. Customers demand that from their suppliers to benefit from
- Choice: Open, disaggregated and standards-based solutions give customers choice to pick the best product, with no vendor lock-in
- Faster Innovation: Vendors compete with each other at every layer of the disaggregated stack and innovate faster
- Best TCO: Open, disaggregated and standards-based solutions often deliver the best TCO
We can see many examples in IT infrastructure where open, disaggregated and standards-based solutions are much more successful, particularly with top cloud customers. They include
- Applications moved from monolithic architecture to cloud native architecture using microservices delivered using APIs and containers from best-in-class providers
- Servers with proprietary HW & operating systems transitioned to open, disaggregated servers using standard server hardware and open-source Linux OS
- Storage solutions moved from integrated array based architecture to open, disaggregated storage architecture
- Networking solutions moved from integrated switches to open, disaggregated standards-based networking. Customers can pick the best-in-class switch silicon, switch system HW and optics modules independently.
Figure 5: Open, Disaggregated and Standards-based Solution across the IT Infrastructure
Open & Standards-based CPO Model
As customers move to higher performance/scale networking solutions, we expect them to complement current switches that take pluggable optical modules with co-packaged optics-based solutions.
As we showcased earlier with IT infrastructure, customers also expect to get open, disaggregated and standards-based solutions for co-packaged optics solutions. They want choice to pick the best product/vendor, faster innovation and best TCO.
In a CPO solution, customers expect to pick the best switch silicon in a system, and be able to use it with the best optics tiles/chiplets from a co-packaged optics vendor. Interoperability between switch silicon and optics tiles/chiplets from different optics companies is critical for the customer. In order to achieve that, solutions need to be built using an open and standards-based approach.
There are multiple industry and customer driven initiatives underway to achieve that goal.
- Two cloud titans, Facebook and Microsoft, started collaborating in 2019 to promote open standards for adoption of co-packaged optics with the Co-packaged optics (CPO) collaboration effort
- There are industry standardization initiatives being driven through OIF and COBO
The goal for these efforts is to develop open and standards-based solutions that are inter-operable, such that customers can select best of breed products in a CPO solution.
At Innovium, we firmly believe that an open, disaggregated and standards-based approach is the right choice for customers looking for co-packaged optics solutions. We welcome and support ecosystem partnerships and industry initiatives to further that cause and are actively driving innovative solutions in this area.
If you are architecting modern data-centers, please contact us at: [email protected] to learn more about our breakthrough networking solutions, including our plans for co-packaged optics (CPO).