This security vulnerability that can be easily reproduced with most X.Org Servers on Linux was discovered by The Invisible Lab (a computer security research firm) and after privately reporting it to the X.Org team back in June was now detailed today in the company's blog and this formal security paper. Below is the summary provided within this paper entitled "Exploiting large memory management vulnerabilities in Xorg server running on Linux."
A malicious authenticated client can force Xorg server to exhaust (or fragment) its address space. If running on Linux, this may result in the process stack top being in an unexpected region and execution of arbitrary code with server privileges (root). x86 32 and x86 64 platforms are affected, others most probably are affected, too.
Note that depending on the system configuration, by default local unprivileged users may be able to start an instance of Xorg server that requires no authentication and exploit it. Also if a remote attacker exploits a (unrelated) vulnerability in a GUI application (e.g. web browser), he will have ability to attack X server.
In case of a local attacker that can use MIT-SHM extension (which is the most likely scenario), the exploit is very reliable. Identifier CVE-2010-2240 has been reserved for the underlying issue (Linux kernel not providing stack and heap separation). This issue has been known for at least five years.
The good news is that this issue is now corrected in the stable 188.8.131.52, 184.108.40.206, and 220.127.116.11 Linux kernel releases (along with the upstream Linux 2.6.36 kernel code) after it was corrected in the past few days via a number of patches. For those running a non-patched kernel it's also possible to make the system less vulnerable to the exploit by disabling the MIT-SHM (MIT Shared Memory) X extension, which when enabled allows the X Server to exchange data between the client and server using shared memory.
Of course, if the X Server wasn't running as root but rather an unprivileged local user, the vulnerability of a GUI application taking advantage of this bug could possibly have been adverted -- at least in terms of exploiting the fundamental kernel memory bug through X. Fortunately, thanks to kernel mode-setting this is now becoming a possibility. The X Server has traditionally had to run as root since X drivers have had to bang the hardware directly, but with these activities being moved into the Linux kernel DRM drivers, there isn't a real need for X to be root any longer. The mainline Linux kernel supports kernel mode-setting for the open-source Intel, ATI, and Nouveau (NVIDIA) drivers and is used by default on most recent desktop Linux distributions.
While most distributions now leverage KMS when using the open-source Intel/ATI/NVIDIA drivers, not many of them at this point are taking advantage of a root-less (in the sense of not not running as the root user) X Server. There's still a few items for the distribution vendors to address such as handling multi-user sessions while still having the X Server running as a non-root user and permission issues with a few areas of X. Though if you are using one of the more obscure open-source graphics drivers (Poulsbo, VIA, XGI, etc) or the proprietary drivers (mainly ATI) running the X Server as root is also not a possibility since they don't currently support KMS, and thus these non-kernel drivers need to still talk to the hardware directly. The proprietary NVIDIA driver doesn't implement KMS support, but it can allow the X Server to not run as root assuming the /dev/nvidiaX files have appropriate permissions.
There's been work towards making the X Server non-root since the kernel mode-setting work was integrated into the mainline Linux kernel. Moblin 2.0 was the first major distribution to abandon running the X Server as root, which was easy in their case since they just aim for Intel hardware support where using KMS is the only choice, and this non-root usage continues to be the case with the MeeGo distribution. Ubuntu is working towards a root-less X Server for Ubuntu 10.10 (though it looks like some bits may now not land until Ubuntu 11.04), as with other Linux distributions, when a KMS driver is utilized. The security-oriented OpenBSD developers have also been interested in porting the Linux KMS code to their BSD kernel for the less of a security threat created by the X Server not having root privileges.