Numascale Builds A 5184 AMD Core Shared Memory System

[phoronix]

Phoronix: Numascale Builds A 5184 AMD Core Shared Memory System

Our friends at Numascale have assembled the world's largest shared memory system to date: 5184 CPU cores with 20.7 terabytes of shared memory...

http://www.phoronix.com/vr.php?view=MTg0MzE

[gens]

edit:
i wrote a funny message about the nsa
phoronix decided to massacre it...

[peppercats]

Can't wait to get my hands on the fx-518400, AMD sure has this more cores thing perfected.

[Adarion]

> 5184 CPU cores

Oh please could somebody gift me one of these systems? Incl. power supplies.

Why the odd number? I mean, why not 6144? And how many processors (or cores) are busy just controlling and managing the whole thing?

[Del_]

> 5184 CPU cores
Why the odd number? I mean, why not 6144? And how many processors (or cores) are busy just controlling and managing the whole thing?

There are 6*6*3 nodes with 48 cores each. The 48 cores corresponds to three 16 core opterons. The fourth socket on each node is occupied by the numascale interconnect which uses AMD's hyper transport to communicate with the CPUs. In effect, for the three CPUs on a node the whole rest of the cluster looks like the fourth CPU. Quite iffy, don't you think?

[ales-76]

> 5184 CPU cores

Oh please could somebody gift me one of these systems? Incl. power supplies.

Why the odd number? I mean, why not 6144? And how many processors (or cores) are busy just controlling and managing the whole thing?

No processors/cores are dedicated to 'manage the whole thing', the machine is cache-coherent NUMA, running single OS image. Kernel is managing all the resources, the same way it does when running on a single server. The network in between the nodes is transparent to the OS, from its perspective it's one big NUMA machine.

The odd number of cores is because the NumaConnect adaptor is based on HyperTransport, and since few servers today are sold with HTX slot, the adaptor is connected to one CPU socket via 'pick-up module'. So in each node there are three Opterons + one pick-up module - check the link for a picture.

http://www.numascale.com/numa_pictures/N323-in-IBM3755.jpg

At any rate, it looks like a relatively inexpensive way to build an HPC monster. I wonder how it stacks up against more expensive systems like SGI UV.

[DrewDahl]

UV

I'm curious how much memory it can actually support. A UV2000 can do 64TB with 5120 cores... curious how it compares.

[ales-76]

I'm curious how much memory it can actually support. A UV2000 can do 64TB with 5120 cores... curious how it compares.

Hard to say how well it scales in practice, but theoretical limits are 256TB in total (48bit address space) and 240GB per node (with 4GB cache and 8GB tag memory per card). So with 108 nodes and 5184 cores the maximum would be 25TB of total memory. Provided that the nodes actually support that much RAM. Some AMD servers are limited to 192GB of RAM, which is what they seem to use - 192GB *108 = 20.7TB