Windows maybe? That seems to be the consensus among the people I've heard selling (and justifying buying) RAM these days
Personally I'm happy with 2GB and don't notice any heavy swap activity.
If you're not doing any multimedia editing, the only other legitimate use for a large amount memory that I can think of is virtualisation (running one or more Xen/VBox/KVM etc. guests on your computer).
Some games may benefit from more memory, but I doubt you'd notice the difference even with most modern games.
EDA (e.g. Xilinx/Altera tools, silicon compilers/simulators, etc.) and numerical computing (e.g. MATLAB/Octave) come to mind as likely candidates, although non-engineers probably don't care about such things. Ultimately it depends on the size of your design or data sets, but the footprint can grow pretty fast.
Extra memory helps if your use scenario includes large total size of frequently used files. Unless you have very fast raid array to make read latency acceptable.
So my idea of what memory is required for low latency:
surfing/mail 256M (minimal mem: xfce, midori, etc)
-,,- 512M (gnome, kde, win xp)
office work/gaming 1G
high end gaming 2G
Heavy computer work 4+G (graphics, engineering, data mining, software development, etc)
Of course you can squeeze limits down if you know a bit more about computers and where to find light applications to save memory. But managing low memory system takes time each day ... time is money and memory is cheaper.
PS. I don't even have 1G memory but maybe soon. My computer is about retire soon as I'm currently running dd_rescue /dev/sda2 - | ssh server.local "gzip > sda2_recover.img.gz"
Swap usage doesn't really tell you much with linux. Most programs that can or do or want to use large amounts of ram usually manage memory themselves. Swaps really just a air bag for when things get totally out of control.
Compiling uses huge amounts of ram. Unfortunately I'm no linux guru I just know enough to know that until you put so little ram in a linux computer that it has no choice it won't use swap and if you put a ton of ram in a linux system you have to tweak some programs to get them to use it.
Don't try to run virtual machines without big ram. THAT I KNOW. Wine will even pig out on ram pretty good if it can and in standard linux fashion sip it sparingly if it can't.
I can tell ya though. Ram is cheap. Buy it cause if we lose one more big ram supplier that will stop.