If I run something like atop while the box is thrashing you can see that its used up all its RAM and there is a lot of paging activity. The thing is, compared to other OS's linux tends to try and push pages that haven't been used for a while to disk, and it has two thresholds; dirty_background_ratio and dirty_ratio ; The background one is sort of the percentage of dirty pages in RAM that you have to have before the VM starts flushing them out. The latter 'dirty_ratio' is a higher percentage threshold. If the amount of dirty pages reaches that percentage then the kernel will block processes from executing until the dirty_ratio gets below that percentage. So hitting the higher percentage is kinda irritating.
There's a good article at rackerhacker about these settings, and there is a note at the end about using 1% as the lower threshold and 100% as the higher one (with the caveat that you really should have a good power supply and RAID card). It might seem weird to have 1% as the lower threshold, as this logically increases disk activity, but with a memory constrained machine you want to start pushing dirty pages out sooner, and having the higher threshold as high as possible means you're telling the OS "Please don't block processes unless its an absolute disaster".
So me, attempting to be marginally conservative tried 3% and 80% for dirty_background_ratio and dirty_ratio respectively and hey it does make quite a difference. A lot of people tend to play with the swappiness setting too, but I've let it as the default of 60%
This article is also good
