Investigating Cavium's ThunderX: The First ARM Server SoC With Ambition
by Johan De Gelas on June 15, 2016 8:00 AM EST- Posted in
- SoCs
- IT Computing
- Enterprise
- Enterprise CPUs
- Microserver
- Cavium
When is a worthy alternative to Intel's Xeon finally going to appear? That is the burning question in the server world. If PowerPoint presentations from various ARM-based SoCs designers released earlier this decade were to be believed, Intel would now be fighting desperately to keep a foothold in the low end server market. But the ARM SoCs so far have always disappointed: the Opteron A1100 was too late, the X-Gene 1 performed poorly, consumed too much power, and Broadcomm's Vulcan project is most likely dead. This Ubuntu page is an excellent illustration of the current state of the ARM server market:
Discontinued products, many announced products which do not even appear on this page (we are in the middle of 2016, after all), and despite the fact that there is an ARM Server Base System Architecture (SBSA) specification, every vendor has its own installation procedure. It is still a somewhat chaotic scene.
Meanwhile, Intel listened to their "hyperscaler customers" (Facebook, Google...) and delivered the Xeon D. We reviewed Intel's Broadwell SoC and we had to conclude that this was one of the best products that Intel made in years. It is set a new performance per watt standard and integrated a lot of I/O. The market agreed: Facebook's new web farms were built upon this new platform, ARM servers SoCs were only successful in the (low end) storage server world. To make matter worse, Intel expanded the Xeon D line with even higher performing 12 and 16 core models.
But losing a battle does not mean you lose the war. Finally, we have a working and available ARM server SoC which has more ambition than beating the old Atom C2000 SoC. In fact, Cavium's ThunderX SoC has been shipping since last year, but you need more than silicon to get a fully working server. Firmware and kernel need to get along, and most libraries need to be compiled with platform-specific optimizations. So the quality assurance teams had a lot of work to do before Cavium could ship a server that could actually run some server software in a production environment. But that work has finally been finished. Cavium send us the Gigabyte R120-T30 running Ubuntu 14.04 server with a ThunderX ready Linux kernel (4.2.0) and ThunderX optimized tools (gcc 5.2.0 etc.).
Cavium?
Who is Cavium anyway? Even for those working in the enterprise IT, it is not a well known semiconductor company. Still, Cavium has proven itself as fabless network/security/storage and video SoC designing company. The company based in San José counts IBM, Juniper, Qualcomm, Netgear, Cisco among its customers.
With a net revenue of about $400 million, Cavium is about one-tenth the size of AMD. But then again, Cavium either reports small losses or smal profits despite heavy investments in the new ARMv8 project ThunderX. In other words, the company's financials look healthy. And Cavium did already design a 16-core MIPS64 Octeon Network Service Processor (NSP) back in 2006. So Cavium does have a lot of experience with high core count SoCs: the network processor Octeon III CN78xx has 48 of them.
Handling server applications is of course very different from network processing. A large amount of independent network packets creates a lot of parallelism, and more complex computation can be offloaded to co-processors. Still, Cavium is the first vendor that delivers an ARMv8 server chip with an impressive core count: no less than 48 cores can be found inside the ThunderX die.
To keep the design effort reasonable, Cavium based their first ARMv8 server processor, ThunderX, on the Octeon III CN78xx. We described this in more detail here, but the main trade-off is that Cavium used a relatively simple dual issue core. As a result, single threaded performance is expected to be relatively low. On the opposite side of the coin however, it is the first ARM SoC that has claimed throughput numbers in the realm of the best Xeon D and even midrange Xeon E5, instead of competing with the Atom C2000. It is the most ambitious ARMv8 SoC that has made it into production.
All of this gives us plenty of reasons to put the Cavium ThunderX through paces. And throwing in the latest Supermicro boards with the latest 12 and 16 core Xeon-Ds made it a lot more interesting ...
82 Comments
View All Comments
JohanAnandtech - Wednesday, June 15, 2016 - link
Good suggestion. I have been using an ipmi client to manage several other servers, like the IBM servers. However, such a GUI client is still a bit more userfriendly, ipmi commands can get complicated if you don't use them regularly. The thing is that HP and Intel's BMC GUI are a lot easier to use and more reliable.fanofanand - Wednesday, June 15, 2016 - link
I think you may have an inaccurate figure of 141 at idle (in the graph) for the Thunder. "makes us suspect that the chip is consuming between 40 and 50W at idle, as measured at the wall"JohanAnandtech - Wednesday, June 15, 2016 - link
If you look at the Column "peak vs idle", you see 82W. At peak, we assume that a 120W TDP chip will probably need about 130W. 130W - 82W (both measured at the wall) = 50W for the SoC alone at idle measured at the wall, so anywhere between 40-50W in reality. My Calculation is a "guestimate", but it is clear that the Cavium chip needs much more in idle than the Intel chips.(10-15W) .djayjp - Wednesday, June 15, 2016 - link
Many spelling/grammar issues here. It impacts readability. Please read before posting.djayjp - Wednesday, June 15, 2016 - link
That is to say in the article.mariush - Wednesday, June 15, 2016 - link
These guys are already working on ThunderX2 (54 cores, 3 Ghz , 14nm , ARMv8) and they already have functional chips : https://www.youtube.com/watch?v=ei9uVskwPNEMeteor2 - Thursday, June 16, 2016 - link
It's always jam tomorrow, isn't it? Intel is working on new chips too, you know.beginner99 - Wednesday, June 15, 2016 - link
It loses very clearly in performance/watt to Xeon-D. In this segment the lower price doesn't matter in that case and the fact that it has a process disadvantage doesn't matter either. What counts is the end result. And I doubt it would cost $800 if made on 14/16nm. I mean why would anyone buying this take the risk? Safer bet to go with Intel also due to more flexible use (single and multi threaded). The latency issue is mentioned but downplayed.blaktron - Wednesday, June 15, 2016 - link
So downplayed. Anandtech desperately wants ARM servers, but its a solution looking for a problem. Big web front ends running on bare metal are such a small percentage of the server market that developing for it seems stupid. Xeon-D was already in development for SANs, they just repurposed it for docker and nginx.Senti - Wednesday, June 15, 2016 - link
Very nice article. I especially liked the emphasis on relations of test numbers and real world workloads and what was problematic during the testing.It would be great to see the same style desktop CPU review (Zen?) form you instead of mix of reprinted marketing hype with silly benchmark numbers dump that plagues this site for quite some time now.
Some annoying typos here and there, like "It is clear that the ThunderX is a match for high frequency trading", but nothing really bad.