Software audio mixing in PulseAudio & JACK is a lot better quality than the hardware mixing in your Creative X-Fi (and hopefully the X-Fi has somewhat better quality sound than the on-board chip—but worse than a professional sound card).
And when it comes to latency, JACK should give you latency of 5ms or likely less (1-2ms?), while PulseAudio should probably be able to go down to about 20ms or less. If you have multi-second latency something is wrong; either your game (or other application that wants/needs low-latency audio) is doing something wrong, or you found a bug in PulseAudio

— this is certainly not how PA is supposed to work.