Unlike most CPU designs AMD’s Ryzen series is not a monolithic design. What we mean is that usually a CPU is one design with all cores and cache, and other miscellaneous parts, built into one large ‘building’. This is the conservative approach that AMD used for generations – and worked well for them. Instead of taking this conservative approach AMD took a page from their APU teams and created a new way of building CPUs. A way that allowed for easily scaling up or down a ‘CPU’ core count, abilities, and basically allows them to simply ‘swap out a floor’ or even add ‘another floor’ instead of creating an entire new ‘building’ from the ground up every time they want to tweak the blueprint.
This certainly is a good thing for consumers and AMD alike as it allowed AMD the luxury of creating a 8-core (Ryzen 7) design and then easily scaling it down to Ryzen 5 (6 cores) and Ryzen 3 (4 cores)… and then simply modifying it to create an entire APU line instead of having to build a new blueprint every time they want to meet the demands of a new market. This in large part is why AMD was able to release a 8-core beast that costs only a smidge more than an Intel 4 core design.
The downside however to using this scalable building block method is that not all cores are interconnected. Instead, much like two separate apparent buildings, each ‘CCX’ block of four cores is internally homogenous but data has to use a ‘highway’ to get from one block to the other. Equally important is this highway is what connects the CCX blocks to the rest of the system.
This highway is what AMD calls Infinity Fabric. To remove the analogy, it is a 256-wide bi-directional crossbar that is used to simultaneously transport data for multiple buses to their final destination (the cores). Basically it is a next generation HyperTransport/FrontSideBus/internal core bus all rolled into one, and in very basic terms can be considered ‘HyperTransport 2.0’. The internal – CCX to CCX – portion is called the ‘Control Fabric’ and the external – Memory bus, etc – portion is called the ‘Data Fabric’. This multi-role responsibility is why it best described as a crossbar and not just bus per say… as it is multi-layered and interconnected bus much like a crossbar interconnector in networking architecture.
For those familiar with Intel designs, think of Infinity Fabric as AMD’s version of ‘uncore’ or ‘System Agent’ on steroids… as this is the ‘data highway’ that connects the various major system components to the CPU (including the memory controller) and the 4-core CCX blocks to each other. Yes it is a big deal, and yes it is a critical component to Zen design. In fact you can say it is the key concept that allows Zen – and not just Ryzen – to be so scalable.
This linking of RAM to Infinity Fabric speed is most likely a major reason why AMD Ryzen CPUs (and probably APUs) have a tough time handling speeds in excess of DDR4-2666 with 32GB kits… as the Infinity Fabric is being overclocked right along with the memory. This however to us means that the faster the RAM the more efficient the CPU is going to be.
As one of its jobs is to tie each CCX together and allow them to talk to directly to each other, in certain instances one entire block of cores could be waiting on data to be delivered from the other block and cannot continue until it gets that data. Thus the less cycles that are wasted on waiting for data to ‘cross the highway’… the better the overall performance. Now AMD in a rather interesting move linked the speed this connector works at to the speed of the RAM. Specifically, it runs at the exact same speed as the integrated memory controller. Yes, by ‘overclocking your ram’ you are actually overclocking this interconnect at the same time.
So let’s talk specifics. When the Integrated Memory Controller’s speed is set to DDR4-2133 this means the Infinity Fabric is running at 1066.5Mhz (as that is the actual clock speed the DDR4 RAM is running at). When using DDR4-2400 it is running at 1200Mhz, and when dealing with DDR4-2666 RAM it is running at a ‘blistering’ 1300Mhz. Taking one extreme to the other this is a 21.9% speed – and potential performance – difference. To put this in more understandable terms when using DDR4-2133 ram the Infinity Fabric is able to handle (an estimated) 68.26 gigabytes of data per second, whereas at DDR4-2666 it is able to push a whopping 83.2GB of data per second. This however is a lot less than the (estimated) 200GB/s that each core in a CCX block can share data with the others cores in the same CCX block. So yes this is a potential bottleneck and one that may help boost overall performance.
Of course, just because the Infinity Fabric can handle 83.2GB/s (at 2666) instead of 68.3GB/s (2133) does not mean it actually needs that extra speed. To make an(other) analogy if you have a ten-lane highway but only 6 cars are on the road… those extra lanes are not going to matter – as everything is humming along with no traffic jams or slowdowns. Basically if the two CCXs are not wasting enough cycles, and the RAM nor PCIe buses are not a bottleneck, the performance differences will be only noticed in synthetic tests and not the real world.
This is the crux of the matter and how this article came about… as there is a lot of synthetic data proving why X is better than Y… there is very little in the way of tangible real world results proving this potential issue is ‘real’ or not.
Before we move on the actual testing phase, please do not take any of this out of context. We are not being apologists for AMD here. AMD should never have linked the Infinity Fabric to the IMC speed. In all likelihood this was done so as to simplify the design and allow for easier building of new consumer ‘CPUs’ or ‘APUs’ based on what at its heart is a enterprise design – where Infinity Fabric is actually a very sophisticated answer to multi-socket interface. In other words, it was a design hack (in the original meaning of the term – and less forgiving people may go as far as saying ‘kludge’) that may come back to haunt AMD in the long-term… or not. Nothing says that as this new technology matures that ‘Zen 2.0’ will be as limited. It is possible that future Zen based models are capable of having the Infinity Fabric combined with the IMC running at uber-high speeds without issue. In the short term though this is a growing pain, and something consumers thinking about building a Ryzen based system need to be aware of and know how big – if at all – this issue really is.
