Intel Advanced Matrix Extensions [AMX] Performance With Xeon Scalable Sapphire Rapids

OpenVINO with relevant models continued showing very nice performance gains with Advanced Matrix Extensions found exclusively for now on Sapphire Rapids processors.

Beyond the significant gains, in most scenarios meant slightly lower power consumption compared to taking just the AVX-512 route.

Color me pretty darn impressed with Advanced Matrix Extensions (AMX) and the performance seen so far with oneDNN and OpenVINO. For this machine translation model the performance more than doubled compared to just AVX-512.

From these initial results the Advanced Matrix Extensions (AMX) performance uplift was huge and without any negative power/thermal repercussions like during the early days of AVX-512. AMX is looking pretty darn good from this software state for "launch day" with oneDNN and OpenVINO. It will certainly be interesting to see how the software support around AMX evolves and how quickly more relevant industry software begins taking advantage of AMX for faster BF16/INT8 with the tile matrix multiply unit.