Doesn't this also mean the display server gets 'less important' because it doesn't do that much any more, at least compared to X11?

On one side it talks to established driver+kernel APIs, and on the other to toolkits like Qt for applications (very simplified, I guess). And as you say, Mir would probably do the same thing. So I'm still not conviced about all the fragmentation doomsday talk. One would probably be able to switch display servers like today we switch DEs?