Why do people gripe about Ubuntu's speed? Its not a fast distro and was never meant to be, its one that is easy to use.
If you want speed you have to
1) Compile your own kernel and optimize it for your processor.
2) look at what drivers you need and build as many as you can into the kernel (i.e. not as a module)
3) leave out unneeded stuff from the kernel
4) Compile your core libraries and optimize it for your processor.
5) Compile your key programs and optimize them for your processor.
6) eliminate as many unneeded background processes as possible.