It isn't really against NVLink, though it may partially be a reaction to it. It is more in opposition to CCIX. It seems Intel chose not to allow 3rd party accelerators faster access to their processors and the system memory, probably because they thought doing so would help NVIDIA more than it helped themselves. But there is too much of a threat against them now to withhold it any longer. If AMD, IBM, and ARM-based servers all offer such support it would give companies a good reason to consider processors from those companies/architectures over Intel processors.
Too bad that AMD, ARM, IBM all together have not a great momentum in IT industry. The bulk of the market is in Intel solid hands. So i can see CCIX disappear in the long run. If Google and Microsoft are with CXL, this is enough to give a strong hit to CCIX.
Mellanox is more rack to rack, this is CPU to accelerator/GPU (probably their own at first). Different uses, different market. But sounds very much like AMDs infinity fabric, which can also run across PCIe links.
Thank you. I was waiting to see somebody mention Infinity Fabric. At least at a superficial level, this sounds to me a lot like what AMD has been doing with that.
Thanks Ian! Two questions: 1. What data transfer speeds are required and supported by CXL? 2. As this requires PCIe 5.0 or higher, what is the earliest CXL-linked devices will even be available? Thanks!
It has been shipping in the POWER9 since late 2017. High end IO devices have been supporting it but with only a handful of host platforms released to date supporting it, there hasn't been much traction. Intel's migration to 10 nm was also supposed to bring PCIe 4.0 to the mainstream. AMD is shipping PCIe 4.0 host devices later this year.
My understanding is that PCIe 4 and 5 are both coming out this year, with PCIe 4 being consumer focused and PCIe 5 being data center focused. There really aren't any consumer workloads that would benefit from a quadrupling (3.0 -> 4.0 -> 5.0) of bandwidth at this point.
*edit : PCIe 4.0, as others have noted, is already out. I meant that both will be available this year and 4.0 should be available on mainstream AMD and maybe Intel consumer boards this year.
Actually there is a consumer use-case that will rapidly take advantage of faster PCIe. M.2 slots. They're kinda short in the lanes department. I really wish M.2 was designed to be more robust in terms of lanes, power, and clearance specs. Also wish they didn't have SATA as part of the spec, at all.
Anyway, PCIe 4.0 is still beneficial even if it's short-lived. If you buy a board mid-2019 with PCIe 4, as opposed to one with 3.0 only, you still have more bandwidth available for a PCIe 5 device. That includes the aforementioned M.2 devices, once 4/5 models hit the market. Obviously PCIe 5 would be better to have, but it's a matter of what's the best thing available when you build your rig.
Both PCIe 4.0 and 5.0 are already implemented in Data centers and servers, mainly the former one.
In consumer, PCIe 4.0 will start this year from AMD, then maybe NV & Intel will follow. SSD makers already started working on PCIe 4.0 based SSD controllers to make use of the extra bandwidth and speed.
PCIe 4.0 requires more strict requirements than PCIe 3.0, that's why it could work on some of the current motherboard designs, so the same design can be used also for PCIe 4.0, thought it could only be for closer to the CPU connections, like only for the closest PCIe slots and to the chipsets, further distances will require additional chips (re-clockers).
PCIe 5.0 on the another hand is another thing, it even requires new motherboard materials, stricter designs for interference, distances, routes, etc.. While this is fine for data centers, these changes adds cost which is harder for consumer grade. So while PCIe 5.0 will be ready soon after PCIe 4.0, the cost will add additional barrier here. We might see PCIe 5.0 takes few more years after PCIe 4.0 in consumer products, or we might see a hybrid PCIe 5.0 & 4.0 implementation to cut costs and so on, like we already now have motherboards with both PCIe 3.0 and PCIe 2.0, we might see motherboards with both PCIe 4.0 and 3.0 in the beginning. Then after few years we will see motherboards with both PCIe 5.0 and PCIe 4.0.
Thanks Ian! I am a bit puzzled, though: one of the slides shown says " CXL specification 1.0 available now". I guess that's Intel speak for "we know what they are, but we won't tell you what they are or where to find them. " Open standard indeed!
It's probably a couple of years off, whilst CCIX (albeit PCIe 4.0) is out now.
This screams of Not In Here syndrome by Intel. It looks like they will be creating their own ecosystem for their own products - FPGAs, CPUs, GPUs and so on. In the meantime CCIX will also move to PCIe 5.0 and be the non-Intel ecosystem (NV-LINK aside).
Can we at least be glad that they're sticking with PCIe signalling and connectivity? At least that leaves some hope of still plugging big iron accelerators into a desktop PC.
Anandtech is turning into the poster child of ad blockers. I don't use one, because I don't mind site owners getting some money, but having the site move from one annoying auto-running video to two, one of which is an ad which can't be even paused, is too much.
Ian, I love Anandtech -- the depth and breadth of the content your team creates is fantastic. However, I have to agree with ET: the adverts are getting ridiculous. I captured your home page, once with uBlock and once without. The uBlock page transferred 2.3MB of data. With the adblocker turned off it was 42.4MB. That's 18.5:1 -- and I didn't even get a video ad.
Please tone it down a bit and I'll gladly whitelist, as I'm sure many others.
Unfortunately editorial has no control over the ads - both type and placement. It's all on our publisher. I don't get a say, although I try and make noise. I can report ads that break the site, or that are wholly inappropriate, but that's tackling specific advertisers, not the placement/arrangement.
Please let your publisher know that I love AT and it's one of my favourite websites, but there is no way I can ever browse it without a strict ad blocker in place. I hate that I can't support AT's writers and editors who do such fantastic work. If there was a reasonably priced monthly subscription program similar to Ars Technica Pro which would allow me to support the website and not be served ads, I'd love that.
Agreed 100%! Ars is $50 per year and you get a Yubikey when you commit for the 12 month period. I'm happy to commit to paying that should the option become available on AT. Or make it $30 per year without any throw-ins. Either way, I'm in.
I don't know what you consider an ad. One is a video of 'how to choose an SSD / CPU / whatever', one is an ad. Having videos run automatically is very distracting, and when one can't be paused is just too annoying. I'm fine with text ads, though.
Wow, after all this there's a second video advert now? I guess the answer to people saying, "Hey, that video in the middle of the article is too much, please find a different way," was to flip the bird and double down.
Had that in the past. I guess the ad server eventually got that I'm less likely to close the page outright if I see pictures of gadgets and good looking women.
Be interesting when it has been investigated a bit. For instance how much does it affect the internal coherency protocol of a chip? Would there for instance be some nasty interaction with Arm's latest interconnect where they probably took CCIX into consideration? Or will it be like GenZ that the standards will probably eventually converge?
Is there a time table for some kind of subscription to Anandtech?
I know a decent chunk of people would be willing to pay a small fee each month to avoid ads and still support the site (and use less battery and data).
So where does AMD's infinity fabric fit in the whole picture? Is this cxl competing for the same space, or is it aiming for a different usecase/market?
The only thing I'm wondering is what does this adds over PCIe 5.0 now or then ?, I can understand NVLink, GenZ that these provides higher bandwidth. But seeing CXL is based on PCIe 5.0 makes me wonder if it will provide the same bandwidth or more bandwidth, in the later case makes sense actually as depending on PCIe 5.0 PHY will lower the costs and being an open standard will make it more popular.
Seems to me it adds cache coherency - meaning if the device wants to read some host memory and the memory is changed in a CPU cache but not written to RAM yet, CPU will know about it and will write its cache to RAM before enabling the transfer; and maybe other way around for accelerators with write-back caches (some TPUs?). The article does not say much about it - as probably the press-release as usually done by non-engineers - people who don't understand that either.
CXL adds asymmetric/biased cache coherency and disaggregated memory pools so that master cpu can give accelerator xpu private access to its own memory pool for big stretches without the coherency monitoring.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
48 Comments
Back to Article
coburn_c - Monday, March 11, 2019 - link
Do you think Intel had this in the can waiting for the Mellanox announcement?blu42 - Monday, March 11, 2019 - link
Clearly it's just a coincidence..BTW, CXL announcement seems better positioned against NVLink and CCIX.
Yojimbo - Monday, March 11, 2019 - link
It isn't really against NVLink, though it may partially be a reaction to it. It is more in opposition to CCIX. It seems Intel chose not to allow 3rd party accelerators faster access to their processors and the system memory, probably because they thought doing so would help NVIDIA more than it helped themselves. But there is too much of a threat against them now to withhold it any longer. If AMD, IBM, and ARM-based servers all offer such support it would give companies a good reason to consider processors from those companies/architectures over Intel processors.Gondalf - Tuesday, March 12, 2019 - link
Too bad that AMD, ARM, IBM all together have not a great momentum in IT industry. The bulk of the market is in Intel solid hands. So i can see CCIX disappear in the long run.If Google and Microsoft are with CXL, this is enough to give a strong hit to CCIX.
Gondalf - Tuesday, March 12, 2019 - link
Absolutely, especially with Huawei and Cisco around.Billy Tallis - Monday, March 11, 2019 - link
OCP Summit is this week, so it's a good time for enterprise/datacenter tech announcements.Haawser - Monday, March 11, 2019 - link
Mellanox is more rack to rack, this is CPU to accelerator/GPU (probably their own at first). Different uses, different market. But sounds very much like AMDs infinity fabric, which can also run across PCIe links.mode_13h - Monday, March 11, 2019 - link
Thank you. I was waiting to see somebody mention Infinity Fabric. At least at a superficial level, this sounds to me a lot like what AMD has been doing with that.levizx - Monday, March 11, 2019 - link
It's more like the Data Fabric part. Not really a direct competitor.eastcoast_pete - Monday, March 11, 2019 - link
Thanks Ian! Two questions: 1. What data transfer speeds are required and supported by CXL? 2. As this requires PCIe 5.0 or higher, what is the earliest CXL-linked devices will even be available? Thanks!A5 - Monday, March 11, 2019 - link
IIRC, PCIe 5.0 PHYs are supposed to hit in 2020. It isn't going to be a long gap like PCI 3 -> 4.p1esk - Monday, March 11, 2019 - link
Wait, PCIe 5 next year? What happened to PCIe 4?Kevin G - Monday, March 11, 2019 - link
It has been shipping in the POWER9 since late 2017. High end IO devices have been supporting it but with only a handful of host platforms released to date supporting it, there hasn't been much traction. Intel's migration to 10 nm was also supposed to bring PCIe 4.0 to the mainstream. AMD is shipping PCIe 4.0 host devices later this year.p1esk - Monday, March 11, 2019 - link
So basically you're saying we *might* see PCIe 4 "later this year", but PCIe 5 is coming next year?mode_13h - Monday, March 11, 2019 - link
AMD already announced (limited) PCIe 4.0 support in (some) *existing* AM4 motherboards!mode_13h - Monday, March 11, 2019 - link
PCIe 4.0 is coming to an AMD 7nm CPU near you!mode_13h - Tuesday, March 12, 2019 - link
Also, AMD has professional/datacenter GPUs based on 7 nm Vega that support it. It'll be disabled in Radeon VII, however.sorten - Monday, March 11, 2019 - link
My understanding is that PCIe 4 and 5 are both coming out this year, with PCIe 4 being consumer focused and PCIe 5 being data center focused. There really aren't any consumer workloads that would benefit from a quadrupling (3.0 -> 4.0 -> 5.0) of bandwidth at this point.sorten - Monday, March 11, 2019 - link
*edit : PCIe 4.0, as others have noted, is already out. I meant that both will be available this year and 4.0 should be available on mainstream AMD and maybe Intel consumer boards this year.Alexvrb - Tuesday, March 12, 2019 - link
Actually there is a consumer use-case that will rapidly take advantage of faster PCIe. M.2 slots. They're kinda short in the lanes department. I really wish M.2 was designed to be more robust in terms of lanes, power, and clearance specs. Also wish they didn't have SATA as part of the spec, at all.Anyway, PCIe 4.0 is still beneficial even if it's short-lived. If you buy a board mid-2019 with PCIe 4, as opposed to one with 3.0 only, you still have more bandwidth available for a PCIe 5 device. That includes the aforementioned M.2 devices, once 4/5 models hit the market. Obviously PCIe 5 would be better to have, but it's a matter of what's the best thing available when you build your rig.
Xajel - Tuesday, March 12, 2019 - link
Both PCIe 4.0 and 5.0 are already implemented in Data centers and servers, mainly the former one.In consumer, PCIe 4.0 will start this year from AMD, then maybe NV & Intel will follow. SSD makers already started working on PCIe 4.0 based SSD controllers to make use of the extra bandwidth and speed.
PCIe 4.0 requires more strict requirements than PCIe 3.0, that's why it could work on some of the current motherboard designs, so the same design can be used also for PCIe 4.0, thought it could only be for closer to the CPU connections, like only for the closest PCIe slots and to the chipsets, further distances will require additional chips (re-clockers).
PCIe 5.0 on the another hand is another thing, it even requires new motherboard materials, stricter designs for interference, distances, routes, etc.. While this is fine for data centers, these changes adds cost which is harder for consumer grade. So while PCIe 5.0 will be ready soon after PCIe 4.0, the cost will add additional barrier here. We might see PCIe 5.0 takes few more years after PCIe 4.0 in consumer products, or we might see a hybrid PCIe 5.0 & 4.0 implementation to cut costs and so on, like we already now have motherboards with both PCIe 3.0 and PCIe 2.0, we might see motherboards with both PCIe 4.0 and 3.0 in the beginning. Then after few years we will see motherboards with both PCIe 5.0 and PCIe 4.0.
eSyr - Tuesday, March 12, 2019 - link
It has been available for almost two years on the market, what else is with it?Ian Cutress - Monday, March 11, 2019 - link
1. Not Specified2. Not Specified
It's more of a 'come out of stealth' type announcement. The answers to the question 'when and how much' are often the last ones to appear.
eastcoast_pete - Monday, March 11, 2019 - link
Thanks Ian! I am a bit puzzled, though: one of the slides shown says " CXL specification 1.0 available now". I guess that's Intel speak for "we know what they are, but we won't tell you what they are or where to find them. " Open standard indeed!psychobriggsy - Monday, March 11, 2019 - link
It's probably a couple of years off, whilst CCIX (albeit PCIe 4.0) is out now.This screams of Not In Here syndrome by Intel. It looks like they will be creating their own ecosystem for their own products - FPGAs, CPUs, GPUs and so on. In the meantime CCIX will also move to PCIe 5.0 and be the non-Intel ecosystem (NV-LINK aside).
mode_13h - Monday, March 11, 2019 - link
The term I think you mean is "Not Invented Here".Can we at least be glad that they're sticking with PCIe signalling and connectivity? At least that leaves some hope of still plugging big iron accelerators into a desktop PC.
ET - Monday, March 11, 2019 - link
Anandtech is turning into the poster child of ad blockers. I don't use one, because I don't mind site owners getting some money, but having the site move from one annoying auto-running video to two, one of which is an ad which can't be even paused, is too much.Ian Cutress - Monday, March 11, 2019 - link
There should only be one video ad. If you see a second, drop an email to Ryan with a screenshot and any details you have.johnnycanadian - Monday, March 11, 2019 - link
Ian, I love Anandtech -- the depth and breadth of the content your team creates is fantastic. However, I have to agree with ET: the adverts are getting ridiculous. I captured your home page, once with uBlock and once without. The uBlock page transferred 2.3MB of data. With the adblocker turned off it was 42.4MB. That's 18.5:1 -- and I didn't even get a video ad.Please tone it down a bit and I'll gladly whitelist, as I'm sure many others.
Ian Cutress - Monday, March 11, 2019 - link
Unfortunately editorial has no control over the ads - both type and placement. It's all on our publisher. I don't get a say, although I try and make noise. I can report ads that break the site, or that are wholly inappropriate, but that's tackling specific advertisers, not the placement/arrangement.aryonoco - Monday, March 11, 2019 - link
Please let your publisher know that I love AT and it's one of my favourite websites, but there is no way I can ever browse it without a strict ad blocker in place. I hate that I can't support AT's writers and editors who do such fantastic work. If there was a reasonably priced monthly subscription program similar to Ars Technica Pro which would allow me to support the website and not be served ads, I'd love that.johnnycanadian - Tuesday, March 12, 2019 - link
Agreed 100%! Ars is $50 per year and you get a Yubikey when you commit for the 12 month period. I'm happy to commit to paying that should the option become available on AT. Or make it $30 per year without any throw-ins. Either way, I'm in.SirPerro - Wednesday, March 13, 2019 - link
Holy shit. As if 2.3MB for a web page wasn't massive enough.ET - Tuesday, March 12, 2019 - link
I don't know what you consider an ad. One is a video of 'how to choose an SSD / CPU / whatever', one is an ad. Having videos run automatically is very distracting, and when one can't be paused is just too annoying. I'm fine with text ads, though.ET - Monday, March 11, 2019 - link
Installed AdBlock for the first time, set it to block only Anandtech. Sad, but necessary. At least Anandtech is readable now.PeachNCream - Monday, March 11, 2019 - link
Wow, after all this there's a second video advert now? I guess the answer to people saying, "Hey, that video in the middle of the article is too much, please find a different way," was to flip the bird and double down.mode_13h - Monday, March 11, 2019 - link
The carnival freak show that's emerging from the ads right above the comments is what's pushing me close to the edge.ET - Tuesday, March 12, 2019 - link
Had that in the past. I guess the ad server eventually got that I'm less likely to close the page outright if I see pictures of gadgets and good looking women.willis936 - Monday, March 11, 2019 - link
Another oneDmcq - Monday, March 11, 2019 - link
Be interesting when it has been investigated a bit. For instance how much does it affect the internal coherency protocol of a chip? Would there for instance be some nasty interaction with Arm's latest interconnect where they probably took CCIX into consideration? Or will it be like GenZ that the standards will probably eventually converge?HStewart - Monday, March 11, 2019 - link
I not sure what you are getting at here with above statement - but you should note the GenZ is included as one of the members on this new standard.I not sure why Facebook and Google are included as members.
GreenReaper - Monday, March 11, 2019 - link
Presumably, because they have huge datacenters and wish to use CXL, and maybe influence it.tk.icepick - Monday, March 11, 2019 - link
Is there a time table for some kind of subscription to Anandtech?I know a decent chunk of people would be willing to pay a small fee each month to avoid ads and still support the site (and use less battery and data).
Thanks!
mayankleoboy1 - Tuesday, March 12, 2019 - link
So where does AMD's infinity fabric fit in the whole picture? Is this cxl competing for the same space, or is it aiming for a different usecase/market?JayNor - Wednesday, March 31, 2021 - link
Looks like AMD might use ccix over infinity fabric, which would provide symmetric coherency among GPUs and CPUs.Xajel - Tuesday, March 12, 2019 - link
The only thing I'm wondering is what does this adds over PCIe 5.0 now or then ?, I can understand NVLink, GenZ that these provides higher bandwidth. But seeing CXL is based on PCIe 5.0 makes me wonder if it will provide the same bandwidth or more bandwidth, in the later case makes sense actually as depending on PCIe 5.0 PHY will lower the costs and being an open standard will make it more popular.peevee - Thursday, March 14, 2019 - link
Seems to me it adds cache coherency - meaning if the device wants to read some host memory and the memory is changed in a CPU cache but not written to RAM yet, CPU will know about it and will write its cache to RAM before enabling the transfer; and maybe other way around for accelerators with write-back caches (some TPUs?).The article does not say much about it - as probably the press-release as usually done by non-engineers - people who don't understand that either.
JayNor - Wednesday, March 31, 2021 - link
CXL adds asymmetric/biased cache coherency and disaggregated memory pools so that master cpu can give accelerator xpu private access to its own memory pool for big stretches without the coherency monitoring.