LRDIMMs, RDIMMs, and Supermicro's Latest Twin
by Johan De Gelas on August 3, 2012 4:45 AM EST- Posted in
- IT Computing
- Intel
- Samsung
- Xeon
- Cloud Computing
- Supermicro
TCO
We did a quick cost calculation based on the following assumptions:
- Your applications are memory limited not CPU limited
- A fully equipped system without memory costs about $8000
- You pay 12 cents per kWh
- The real yearly (and not only in the winter) PUE is about two, or if you don't own the datacenter, you pay twice the electricity costs (24 cents per kWh; datacenter clients often pay higher fees for energy costs)
We did a 3-year TCO calculation, without the management costs. Managing half as many physical servers reduces management costs, but it won't be much. Most of the administration costs depends on how many VMs you have to administer, which is the same in both cases.
| Cost | 32GB LRDIMMs | 16GB RDIMMs |
|---|---|---|
| Number of servers | 1 (2 nodes) | 2 (4 nodes) |
| Price per server | 8000 | 8000 |
| Max RAM per server | 1024GB | 512GB |
| Price per DIMM | 1100 (32GB) | 290 (16GB) |
| Number of DIMMs | 32 | 64 |
| Energy per server (kW) | 0.4 | 0.4 |
| Server cost | 8000 | 16000 |
| RAM investment | 35200 | 18560 |
| Energy costs (3 years) | 2523 | 5046 |
| TCO w/o management costs | 45723 | 39606 |
With the current prices, LRDIMMs cannot justify the cost by hardware alone. But the difference is not that large anymore (about 15%). On a 3-year basis, administering half as many servers could definitely reduce the administration costs, but it is very hard to calculate. It could amount to several thousand dollars though, depending on the local labor costs.
If you are paying a software license per server (Ansys comes to mind) and you are memory starved, 32GB LRDIMMs are very attractive. In all other cases, 16GB RDIMMs are most likely the best choice. Unfortunately, VMware decided to make their larger customers pay per vRAM used, so investing in the largest memory available does not reduce licensing costs at first sight when you are running an ESXi cluster. The situation is entirely different with Microsoft's Hyper-V, KVM and Citrix Xen however. This customer unfriendly licensing could convince quite a few people to look the other way. In that case, LRDIMMs allow you to significantly reduce the response time even when the CPU load is high. Our virtualization benchmarking scenario was among the worst to show off the benefits of larger memory capacities and we still noticed up to 2.5 lower response times and 15% lower power consumption.
Back to the hardware. The iMB buffer technology has very little latency impact, so you should not worry about that. The end conclusion is that LRDIMM is a mature technology that offers 50 to 100% more memory capacity or slightly higher bandwidth, but currently at a cost that is about 3 to 4 times higher. If your application is memory limited and you are paying hefty licenses per server, you should definitely consider it. Also, LRDIMM prices are falling. As we type this, industry sources tell us that the price of 32GB LRDIMMs will soon be around $800. If you are in the market for a new high capacity server, keep an eye on the DIMM pricing.
And what about the Supermicro SuperServers? Well, the Twins are typically not considered for virtualization purposes, but now that the latest ones come with 16 slots and 10Gb Ethernet, they become a worthy cost, space, and energy efficient alternative. Our Twin worked flawlessly with the latest ESXi 5, although with 8 DIMM slots (16 DIMMs total), it is not meant to be a virtualization server. We have good hopes that the SuperServer 6027TR-D70RF+ (32 DIMMs total) will be a very affordable alternative to the tier oone OEM servers out there.
Best of all, most of the Twins can be found on VMware's and other Hypervisors's Hardware Compatibility Lists (HCLs). You can definitely save a lot of money as not only the servers themselves are very attractively priced, but you only also benefit from the fact that you do not have to pay a "vendor specific" premium for high capacity memory.
Facebook
Twitter
RSS
26 Comments
View All Comments
koinkoin - Friday, August 3, 2012 - link
For HPC solutions I like the Dell C6220, dense, and with 2 or 4GB of memory per cpu core you get a good configuration in a 2U chassis for 4 servers.But for VMware, servers like the R720 give you more room to play with memory and IO slots.
Not counting that those dense server don’t offer the same level of management and user friendliness.
JohanAnandtech - Friday, August 3, 2012 - link
A few thoughts:1. Do you still need lots of I/O slots now that we can consolidate a lot of gigabit Ethernets in Two 10GBe
2. Management: ok, a typical blade server can offer a bit more, but the typical remote management solutions that Supermicro now offers are not bad at all. We have been using them for several years now.
Can you elaborate what you expect from the management solution that you won't expect to see in a dense server?
alpha754293 - Friday, August 3, 2012 - link
re: network consolidationNetwork consolidation comes at a cost premium. You can still argue that an IB QDR will give you better performance/bandwith, but a switch is $6k and other systems that don't have IB QDR built in, it's about $1k per NIC. Cables are at least $100 a piece.
If you can use it and justify the cost, sure. But GbE is cheap. REALLY REALLY cheap now that it's been in the consumer space for quite some time.
And there aren't too many cases when you might exceed GbE (even the Ansys guys suggest investing in better hardware rather than expensive interconnects). And that says a LOT.
re: management
I've never tried Supermicro's IMPI, but it looks to be pretty decent. Even if that doesn't work, you can also use 3rd party like logmein and that works quite well too! (Although not available for Linux, but there are Linux/UNIX options available out there as well).
Supermicro also has an even higher density version of this server (4x half-width, 1U DP blade node.)
JonBendtsen - Monday, August 6, 2012 - link
I have tried Supermicro IPMI, works nicely. I can power on/off the machine and let it boot from a .iso image I have on my laptop. This means that in case I have to boot from a rescue CD, then I do not even have to plug a CD drive into the machine. Everything can be done from my laptop, even when I am not in the office, or even the country.bobbozzo - Tuesday, August 7, 2012 - link
Can you access boot screens and the BIOS from the IPMI?For Linux, I use SSH (or VNC server), but when you've got memory or disk errors, etc., it's nice to see the BIOS screens.
Bob
phoenix_rizzen - Thursday, August 9, 2012 - link
Using either the web interface on the IPMI chip itself, or the IPMIView software from SuperMicro, you get full keyboard, mouse, console redirection. Meaning, you can view the POST, BIOS, pre-boot, boot, and console of the system.You can also configure the system to use a serial console, and configure the installed OS to use a serial console, and then connect to the serial console remotely using the ipmitool program.
The IPMI implementation in SuperMicro motherboards (at least the H8DG6/H8DGi series, which we use) is very nice. And stable. And useful. :)
ForeverAlone - Friday, August 3, 2012 - link
Only 128GB RAM? Unacceptable!Guspaz - Monday, August 20, 2012 - link
It starts to matter more when you're pouring on the VMs. With two sockets there, you're talking 16 cores, or 32 threads. That's the kind of machine that can handle a rather large number of VMs, and with only 128GB of RAM, that would be the limitation regarding how many VMs you could stick on there. For example, if you wanted to have a dedicated thread per VM, you're down to only 4GB per VM, which is kind of low for a server.darking - Friday, August 3, 2012 - link
I think the price on the webpage is wrong. or atleast it differs by market.i just checked the Danish and the British webstores, and the 32GB LRDIMMS are priced at around 2200$ not the 3800$ that the US webpage has.
JohanAnandtech - Friday, August 3, 2012 - link
They probably changed it in the last few days as HP as lowered their price to $2000 a while ago. But when I checked, it was $3800