AVX-512 Performance Comparison: AMD Genoa vs. Intel Sapphire Rapids & Ice Lake

First up was running Neural Magic's DeepSparse CPU-based inference engine that is able to leverage AVX-512.

Right off the bat with the NLP Text Classification model, we see Sapphire Rapids' AVX-512 implementation yielding a greater boost from AVX-512 compared to the other CPUs. The Ice Lake performance went up by 24% with AVX-512, the EPYC 9654 2P performance went up by 20%, and the new Sapphire Rapids processor enjoyed a 49% boost to the performance thanks to AVX-512.

It was a significant reduction in latency too with the Sapphire Rapids AVX-512 enabled.

While some AI software already has taken advantage of Advanced Matrix Extensions (AMX) or relying upon Intel's oneDNN library, DeepSparse is using neither when searching through their latest development code. It will be interesting to see if eventually they implement AMX support for even greater performance.

While the Xeon Platinum 8490H 2P server was enjoying a greater relative performance lift from AVX-512 being enabled, the AMD EPYC 9654 2P server was generally delivering the best performance overall.

The AVX-512 raw performance uplift on the Sapphire Rapids server was relatively larger than both that of prior generation Ice Lake and AMD's 4th Gen EPYC competition. In a few rare cases the AMD Genoa performance regressed with AVX-512.

For many of the test cases, the Xeon Platinum 8490H CPU peak frequency was similar whether AVX-512 was on or off. That's good news as unlike Ice Lake where where AVX-512 was engaged, the peak frequency was often 100MHz or more lower than the non-AVX-512 run.

Ice Lake with heavy AVX-512 use at times also leads to higher core temperatures.

Across the wide range of tests run with DeepSparse, AVX-512 enabled with Sapphire Rapids was leading to a bigger relative improvement than seen with Ice Lake or AMD Zen 4.

And importantly there wasn't a negative impact on the CPU peak frequency when engaging AVX-512 now with Sapphire Rapids, unlike prior generation Xeon Scalable CPUs.