Setting Up Intel 4th Gen Xeon Scalable "Sapphire Rapids" For Accelerator Use

idxd-config / accel-config

It's simply not a matter of only needing the IDXD kernel driver but there is the user-space elements too. Via Intel's idxd-config on GitHub is the main library and utility for helping to configure and use the DSA accelerators under Linux. The idxd-config code is available via GitHub and of course open-source. Over time we'll likely find idxd-config packaged up by various Linux distributions to make it easier to get started with Intel's accelerators. As noted yesterday, Debian added idxd-config to its archive and Ubuntu 23.04 in turn is set to introduce it too. Over the course of this year it will then be easier to find this library and utility available without having to manually build it from source.

Once installed and prior to use, the accelerators need to be configured. While one can manually write to the sysfs dsa files under /sys/bus/dsa/devices/dsa, it's not intuitive and far easier using the accel-config utility. The accelerators can be organized into groups, workqueues established, and only then can they actually be enabled for use.

The accel-config utility allows for saving and loading JSON-based configuration files for easily managing the accelerator configurations. Since the accelerator configurations are not persistent and need to be set after each boot. For better understanding the accelerator configuration options, Intel does have a few different configuration examples (app_profile.conf is where one can get started).

Additionally, Intel has the DSA Performance Micros repository for helping users/administrators figure out how to configure their accelerators for their desired use-cases. Again, there is also some configs there for showing varying numbers of engines and different numbers of workqueues being enabled. With the accelerator configuration is also a matter of whether it's a dedicated or shared queue as well for allowing the accelerators to be shared among users/clients or not.

Once setup, the DSA workqueues are then exposed under /dev/dsa/wq0.* and those are the character files in turn to be used by the libaccel-config library or if interacting with the DSA workqueue directly. Via standard file permissions you can then modify the /dev/dsa/wq0.* permissions if wanting to grant some users access (or not) to given workqueues.

The accel-config utility wasn't the most intuitive to work (e.g. "accel-config help" not actually being helpful but it's actually "accel-config --list-cmds" for what most people will want to see to get started for the available options) until spending the time and getting the hang of it. The ability to save/load configurations is great. It's unfortunate though there isn't an easy way to streamline this setup/enablement process. Especially for those with single-user servers and planning to just engage one application/workload at a time, it would be nice if there was a easier starter path to onboard new developers/administrators to reduce the initial setup burden. Hopefully with time Intel software engineers will further refine this process, but at least it does offer a wealth of customization ability.

Summing Up The Linux Software Support

You'll want to ideally be on Linux 5.18+ (though 5.6+ work with reduced functionality) for the IDXD kernel driver support, check on your IOMMU options or be aware of the aforementioned workaround if needed, and you'll likely want to have accel-config installed on your system. For now that accel-config will likely need to be built from upstream source but in forthcoming Linux distribution releases will likely become available via the package management systems.

You'll also need to be on a recent GCC or LLVM Clang compiler release within the past two years or so for the Sapphire Rapids support, namely around ENQCMD and MOVDIR64B instruction support.

Beyond the base support, you'll also likely want some additional packages too like the dsa-perf-micros and for IAA usage to have the Intel QPL library.