Hmm, strange results. And as he said the test isn't a real one.
I actually use twm (though it is nearly no use... haha) but it is really small and loads fast. I also use KDE 4 and it doesn't suck so much mem. Also depends on how you compiled things. I'd have to check again but I had really litte memory usage on my fbackup fileserver without X, I ran KDE 3 and 4 on my now dead 512 M RAM laptop and it was actually working. Also had Gnome 2 there which also was running fine.
Admitted I was always using Gentoo. And not pulling in things I do not consider as good or useful (like all that semantic desktop searching and indexing KDE offers). Compiling things with -Os also seems to help, I did this for my smaller machines.