December 19, 2024
Custom, Copper and Cross-Country Connectivity: Eight Big Trends for Marvell in 2024
By
Michael Kanellos, Head of Influencer Relations, Marvell
What happened in semis and accelerated infrastructure in 2024? Here is the recap:
1. Custom Controls the Future
Until relatively recently, computing performance was achieved by increasing transistor density à la Moore’s Law. In the future, it will be achieved through innovative design, and many of those innovative design ideas will come to market first—and mostly— through custom processors tailored to use cases, software environments and performance goals thanks to a convergence of unusual and unstoppable forces1 that quietly began years ago.
- At Marvell AI Day, we announced that custom processors would represent 25% of the AI accelerator market by 2028 and exceed $42 billion. We also announced we had secured four designs with three of the major hyperscalers.
- In July, we unveiled StructeraTM A and Structera X,2 CXL controllers for near memory compute and increasing memory capacity, respectively. Merchant and custom versions of the chips were released at the same time, a reverse of the traditional pattern where custom chips might follow merchant ones by a lengthy lag.
- At OCP, Meta showed off FBNIC, a custom network interface developed in collaboration with Marvell to reduce the time needed to resolve network issues.3 Make that four out of four hyperscalers. See the image below.
FB NIC on display at OFC
- In December, we announced a five-year collaboration that includes supplying custom AI products and other devices for AWS.4 The collaboration also encompasses performing semiconductor design tasks in the cloud, which promises to reduce the time and cost of creating new chips.
- Finally, we ended the year with an alliance for custom high bandwidth memory with Micron, Samsung Electronics, and SK hynix.5 Custom HBM can reduce chip real estate by up to 25% and increase memory capacity by up to 33%.6 Custom HBM runs against industry orthodoxy,7 writes analyst Joseph Byrne, but it could pay dividends.
2. Optical Connectivity Experienced a Cambrian Explosion
Networking consumes around 10% of the power in a traditional server rack and 5% in an AI server rack.8 Connectivity—the copper and optical technology responsible for transmitting data—consume about half of that, so 5% and 2.5%.
Still, data centers are on a quest to reduce power wherever they can. To that end, companies are developing new designs and devices optimized for distance, bandwidth and power.
Aquila, the industry’s first O-band coherent-lite optical DSP,9 provides more performance than traditional PAM4 DSPs for 2 to 20km connections but consumes less power than standard long-reach coherent optical DSPs.
Other concepts that gained momentum in 2024 included transmit-only optical DSPs10 that can reduce module power for mid-length connections by up to 40%, drivers and transimpedance amplifiers for short-reach LPO connections, 3D silicon photonics or SiPho engines11 for simplifying module design, and leveraging 3nm process manufacturing12 to reduce the power of cutting-edge PAM4 optical DSPs for inside-the-data-center connections by over 20%.
In today’s AI clusters with 1,000 XPUs, optical modules outnumber XPUs by a 2:1 ratio. With 100K clusters, it is slated to go to 5:1; see an artist’s rendering of what that might look like below. When clusters hit 1 million, the ratio climbs to 10:1. Expect to see further species of optical devices evolve to meet this challenge.
3. Copper Got Cool
Computers face a bandwidth bind. Today’s PCIe Gen4 links transmit data at 16GB per second between GPUs, CPUs and other chips on a board, and the links can extend for approximately eight inches.
With PCIe Gen 6, coming now, data rates climb to 64GB but distances shrink to around two inches,13 a recipe for thermal hot spots and design headaches. PCIe retimers,14 a new class of chips, extends that to 17 inches. 650 Group estimates that 75% of AI and cloud servers in two years will contain PCIe retimers15: not bad for a device that didn’t even exist a few years ago. PCIe Gen 7—see below—is in development:
Likewise, Active Electrical Cable (AEC) DSPs will extend the reach of 200G/lane optical cables between servers in racks from an impractical one meter to three. Leading cable providers like Amphenol, TE Connectivity and Molex16 have demoed products containing Alaska® P, a 1.6G (8 200G lanes) AEC DSP.
So how did copper pull off this trick? By borrowing SerDes and other technologies from the optical world.
4. Pluggables Went the Distance
Pluggable coherent modules have been taking over more of the market for data center interconnects and long-distance telecommunications networks because they can lower Capex and opex by up to 75%17. Traditional solutions come in the familiar ‘pizza box’ format. Pluggables are about the size of a mozzarella stick.
Where traditional solutions have had the advantage is in distance.
Not anymore. Earlier this year, Marvell, along with Lumentum and Coherent, demonstrated that it is possible to make a network with 800G pluggables from different vendors that can go 500km (Santa Clara to L.A.) and 1000km with probabilistic constellation shaping (Santa Clara to Phoenix). If you crank the speed down to 400G, you can hit 2,500km (Santa Clara to Austin).
5. SONiC Broke the Barrier
SONiC (Software for Open Networking in the Cloud) is the Linux of networking, a vendor-agnostic operating system and software platform for switches to free customers from vendor lock-in and give them the power to optimize their infrastructure as they see fit.
SONiC is now deployed at seven of the top ten cloud service providers18 and growing at 25% per year, as we noted when we announced the commercial availability of Teralynx® 10. And with greater openness often comes innovation.
If you want the full Teralynx 10 experience, we recommend this video from Patrick Kennedy at STH.19
George Hervey at Marvell gives the motherboard tour of a Teralynx 10 switch.
6. Automotive Ethernet Moves Toward a Milestone
Ethernet has been the reigning standard for IT networking for over four decades. Using Ethernet for automotive is a relatively recent phenomenon but it’s growing fast thanks to growing demand for ADAS, infotainment and other smart features. Under current projections, in fact, the total number of ports shipped to car makers will exceed those going to data centers by the end of the decade.20
Those auto ports will be slower on average than their IT counterparts—current ethernet switches being installed by car makers today are in the 10G range—but the volume of data and the urgent need for rapid analysis of data-intensive sources like video cameras will push performance continually upward.
7. Telco Discovered AI
How will wireless carriers take advantage of AI? Initially, it will be to fine-tune their operations to reduce power, space and cost. To that end, we opened the API to the Al/ML core inside OCTEON® 10 to the Linux Foundation’s Datapath Developer Kit and the Apache device library. Now, developers can more easily spin up apps for predictive maintenance and other services.
Nokia included the AI/ML core into its ReefShark chipset and Vodafone is looking into ways to leverage AI21.
8. And Finally, One of the Last On-Prem Apps Got the Green Light to Go Cloud
Hardware security modules (HSMs) are a lesser-known technology, but one that people unknowingly interact with on a daily basis. The 2 billion credit card transactions that take place worldwide per day22 are largely facilitated by HSMs.
But to date, most HSM transactions are performed by on-premise boxes23 managed by the banks and other end-users themselves.
In July, NIST gave FIPS 140-2/3 certification to Marvell LiquidSecurity® 2, a cloud-based HSM that fits on a PCIe card. The upshot: payment specific transactions—40% of HSM traffic—can now be handled in the cloud.
The footprint for moving the cloud also exists. Eight of the ten largest cloud providers have deployed LiquidSecurity products in their infrastructure and developers such as InfoSec, Cryptomathic, and HashiCorp24 are supporting their services on LiquidSecurity.
And now, it’s on to 2025.
1. https://www.marvell.com/blogs/2006-the-twelve-months-that-changed-the-chip-industry.html
6. https://www.marvell.com/blogs/custom-hbm-what-is-it-and-why-its-the-future.html
7. https://xpu.pub/2024/12/12/custom-hbm/
13. https://www.marvell.com/blogs/nine-things-to-remember-about-the-future-of-copper-in-computing.html
15. https://www.marvell.com/blogs/nine-things-to-remember-about-the-future-of-copper-in-computing.html
19. https://www.servethehome.com/inside-a-marvell-teralynx-10-51-2t-64-port-800gbe-switch/
20. https://www.electronicproducts.com/the-road-ahead-for-data-centers-on-wheels/
22. https://capitaloneshopping.com/research/number-of-credit-card-transactions/
23. https://www.marvell.com/blogs/bringing-payments-to-the-cloud-with-fips-certified-liquidsecurit.html
24. https://www.marvell.com/blogs/the-big-hidden-problem-with-encryption-and-how-to-solve-it.html
This blog contains forward-looking statements within the meaning of the federal securities laws that involve risks and uncertainties. Forward-looking statements include, without limitation, any statement that may predict, forecast, indicate or imply future events or achievements. Actual events or results may differ materially from those contemplated in this blog. Forward-looking statements are only predictions and are subject to risks, uncertainties and assumptions that are difficult to predict, including those described in the “Risk Factors” section of our Annual Reports on Form 10-K, Quarterly Reports on Form 10-Q and other documents filed by us from time to time with the SEC. Forward-looking statements speak only as of the date they are made. Readers are cautioned not to put undue reliance on forward-looking statements, and no person assumes any obligation to update or revise any such forward-looking statements, whether as a result of new information, future events or otherwise.
Tags: custom computing, ASIC, AI accelerator processing units, AI, Optical Connectivity, server connectivity, Meta Connectivity, Security, sonic, Silicon Photonics Light Engines, SONIC NOS
