The Xfbdev binary we ship is 646.5 kb. All the XKB things you mention are not needed with Xfbdev.
So we've got ~2.8MB versus 1.6MB for my fully-stripped Xfbdev, which is still a bit of a jump but not half as catastrophic as you make out. Remember also that both of them depend on the 197KB xkbcomp, the 142KB libxkbfile, and an XKB data set which is 3.6MB (!!) unless you go out of your way to strip that down.
It's not a general purpose box usually that is that space conscious or runs Xfbdev. You can often not use gtk, qt, cairo, pixman and folks.
There are still definitely some space and size wins to be squeezed out of Xorg, but it really isn't a massive increase over Xfbdev. Even so, you still have libX11 weighing in at 1.5MB+, and a fairly hefty pixman too. And your clients are likely running GTK+ (bigger than all of them combined), or Qt (even bigger again), with Cairo (pretty damn big).