People must have forgotten about X Toolkit Intrinsics. It's basically what everyone here is proposing (common toolkit with policy and themes implemented by GTK/Qt/etc). It was only used by Motif and maybe Athena widgets. Obviously, it failed.
People must remember that X is equivalent to GDI on Windows. The toolkits + the window manager is equivalent to USER on Windows (basic window management and input demultiplexing are done in USER on Windows, but X on Linux, so that is a key difference). Since Windows presents a little bit more of a unified API and only one toolkit that comes with the system, it appears as though it works differently from X. It doesn't really. And Linux could have chosen to have one toolkit and then this entire discussion would be moot.
I really think the one-piece-at-a-time folks have it right. The overall structure of X is fine. Cruft from the protocol should be removed. Newer, but standard, features should be moved into the core protocol. Xlib should go away. And so on. A total rewrite would be stupid. Even the Windows world has kept the same basic API around for longer than X has been around, despite changes in the rendering system. GDI and USER are still here, even though there is also DWM and WPF. The best path is to provide a way forward and push new software and toolkits to be based on the new features and deprecate the old.
The only thing X really needs right now is man-power. Unfortunately, it seems like it's not top priority. It'd be nice if Canonical, for example, hired a few devs to work just on X. But I doubt that's going to happen. They can barely contribute to the kernel as it is.
Thank you. I was just about to remind everyone about Xt. It was a failure and we should learn from them.
There's the xserver itself, the protocol headers, the client libs, mesa, input and output drivers, fonts, apps, and utils needed for building (libpciaccess, libpixman, etc.). In addition certain parts have external dependencies as well. You only need certain parts depending on what you are building. So if you look at http://cgit.freedesktop.org/ you can see how the xorg related bits are organized. Repos prefixed with /xorg/app are demo X apps that provide various functionality. They are not required to build the xserver. In most cases, the versions installed on your system are fine. Repos prefixed /xorg/font are the old font packages. In general there's no need for these as well. /xorg/util are various utilities used for building and maintaining the code and making releases. The macros package is the most important as it includes the latest m4 macros needed to build the xserver and related things like the drivers. /xorg/driver are the input (mouse, keyboard, etc.) drivers and output (video; ati, intel, s3, etc.) drivers. The input driver provide special handling and options for input hardware. The output provide gpu specific acceleration and in some cases modesetting code. /xorg/lib are the client libraries. Applications that want to use X or a certain extension (Xv, XvMC, Xfixes, etc.) need to have the client libs available. Note that you don't really have to update the client libs to build a newer xserver. /xorg/proto are the protocol headers. These are needed by the xserver and the client libs as they define the protocol for X11 and extensions. /xorg/xserver is the xserver itself. And finally xcb is the new X C binding.
As to the xserver source itself (http://cgit.freedesktop.org/xorg/xserver/tree/), you have the server side implementations of various extensions (Xv, xrandr, render, glx, record, composite, damage, etc.), the rendering code, the input handling code, the acceleration infrastructures (XAA and EXA), common code for modesetting and display handling (edid parsing, i2c) and code for parsing config files and loading drivers.