The console format isn't anywhere near expressive enough. XKB's file format is _too_ expressive and complex for sure, and indeed I'd like to shift to another format. But the console keymaps simply do not let you do quite a number of things that you need for usable keyboard handling on the desktop. Having a new file format would definitely be a win at some stage, but the stark reality is that everyone uses XKB keymaps, and there are a huge number of them that people need to be able to use their computers. Changing file format without a clean conversion is a complete non-starter.
Historically, it had too many esoteric layouts that weren't widely used, but these have almost all been shuffled out to a separate 'extra' ruleset now. Being able to turn those off at install time so they're not installed at all would be a huge boon.
The code can absolutely get a lot smaller. It's been on the backburner for a while as I've been busy with a lot of other stuff, but I'll hopefully get to it at some stage.
You're not going to get away from XKB. Yes, it has its flaws (believe me, many of them), but it's so deeply embedded in the X protocol - and Xlib API/ABI which we can never change - that you cannot have a usable desktop X11 implementation without XKB. For Wayland, my current solution has been to pare down the XKB parser to the bare essentials that both clients and server require for useful and fully-capable keyboard input and then see where that takes us in terms of file format. I've already been able to deprecate quite a few features and remove a fair bit of complexity, and I'm not done yet. But for X11, I'm afraid you're pretty much stuck with XKB as it is from now until the end of days.