Micron's HSE Storage Engine 2.0 Debuts - No Longer Relies On Linux Kernel Modifications

In April 2020 there was the somewhat surprising announcement of Micron announcing their own open-source storage engine designed for SSDs and persistent memory. The Heterogeneous-Memory Storage Engine (HSE) has been redesigned and on Monday debuted in v2.0 form with some fundamental changes.

HSE was started to exploit the performance potential of speedy solid-state drives and persistent memory. The original design of HSE incorporated a Micron-developed Media Pool "MPool" for the Linux kernel to handle object storage. This MPool bypassed conventional file-systems in the name of lower overhead but added further complexity to that storage engine.

As noted in the original announcement on the matter last year, "Until (if) this Mpool kernel module is upstreamed to the Linux kernel will be a major blocker for adoption." They tried last year in wanting to upstream their Mpool Object Storage into the Linux kernel but there wasn't much interest and ultimately the patches didn't move forward.

Now with HSE 2.0, they have done away with the MPool kernel module and this storage engine is strictly in user-space and running atop conventional Linux file-systems.

Shifting away from needing to use a modified Linux kernel for that MPool functionality is one of the big changes for HSE 2.0. Various semantic changes also better align HSE 2.0 with other databases and storage engines for easier porting. Python bindings were also added for HSE 2.0.

Due to these changes, any HSE 1.x software like Micron's modified MongoDB implementation need to be updated for HSE 2.0 compatibility. HSE 1.x is also now deprecated and no longer maintained.

Downloads and more details on HSE 2.0 via GitHub. Now that no kernel changes are required, it will be interesting to see if this drives any new interest in HSE. This storage engine has been on my lengthy TODO list for benchmarking and with HSE 2.0 will hopefully find the time for that soon (premium members, let me know if it's of interest to you).