Diagnosing & fixing unacceptably slow I/O performance

[allquixotic]

That cryptofs is a HUGE overhead, whether the author will admit it or not (the author of such transparent filesystems is usually reluctant to admit that their software is slow, but it is). The overhead is mainly visible when you're reading lots of small files, not one or two big ones. An i3 is a "cheapo" really low-end CPU to be running a cryptofs on, because it's going to take noticeably longer to decrypt data than a faster CPU. I tried a cryptofs on my i7-3820QM and it was so slow that I had to reinstall without it. FWIW my sound menu opens as fast as i can click the button (no noticeable visual delay) without the cryptofs.

You might intuitively think that most programs installed in /usr or /opt wouldn't read from the encrypted filesystem. You'd be wrong. All the "dot" directories (directories beginning with ".") in your home folder are being very frequently read and written to by various programs; web browsers and gnome programs are particularly egregious offenders. Try running an strace on your favorite slow program and grep for "/home/<user>/."

A bargain-basement system to begin with is going to really underperform when you throw a lot of crypto at it. I'm not at all surprised. Hardware damage is thrown out by the passed SMART tests, so all I can conclude is that it's slow because it's a slow system. Unless you've been running the exact same software stack (without grabbing updates) for a long time and it was fine but then suddenly started tanking. That would be different.

My HDD is a HITACHI HTS725050A7E630. The I/O performance doesn't blow me away, but it's certainly about what I'd expect of a typical 7200RPM, 500GB laptop hard drive. Since yours is 5400RPM, latencies will be higher to begin with. Add in the extra effort of a cryptofs and you have a recipe for slow.

Also -- decrypted pages from a cryptofs are cached in RAM, like any other filesystem, to reduce disk activity. Since Ubuntu greedily eats RAM in the Unity environment, it's not going to leave much RAM left over for page cache. Page cache translates directly to high performance, especially when your fs is encrypted. So your slow hdd has to frequently read lots of tiny little files through an encryption layer on a bargain-basement slow CPU and then read them again when they are accessed again because they're constantly getting bumped out of the page cache.

Sounds like your problem is a systemic one; i.e., the reason for the observed result isn't just one single component but a combination of factors due to using cheap components and a high-overhead crypto layer. It's kind of like metabolic syndrome in older/obese people, and how your entire body starts to fail, but treating just one organ doesn't slow down the downward spiral. You have to treat the entire system.

I think you'd have expected performance levels if you:

1. Double your RAM to at least 4GB (even my extremely cheap employer, who has no idea what IT requirements its employees actually have, gives us 4GB)
2. Get rid of the ecryptfs
3. Upgrade your HDD to a 7200rpm model. They have 'em up to at least 750GB but you can find them affordably at the same capacity as you have (500GB).

I'd almost say that upgrading the processor would help too, except that the CPU isn't going to be nearly as busy if it isn't decrypting files, so you might find that the CPU has plenty of spare cycles when removing the ecryptfs.

You can also try, without upgrading any hardware, setting your ext4 filesystem to mount as "data=writeback,nobh" (documented here) -- this will give the filesystem much more flexibility about when to commit to disk, which results in less seeking and less frequent disk writes. Writes will pile up in RAM unless specifically fsync()ed to media or unless there's no more room in RAM. It should keep the disk relatively quiet unless it needs to read something that isn't from the page cache (which again is very likely with only 2GB).

[WSmart]

Power issues can cause massive hard drive lag. I don't know if SMART would catch that. Just a thought.

The power supply itself can be an issue, but also molex connections that have been plugged in and out a number of times will loosen up so that the metal parts in the plug won't make good contact -very common issue, but generally associated with system freezes as well as lag.

We had a computer here that was really bad. It would take like twenty minutes for the hard drive light to settle down when loading winblows. You couldn't do anything for that long, and it was like any button you hit caused the hard drive light to come on full load for minutes and the system wouldn't respond. It was that way for years. Somebody over here just to like complain I think. But it finally broke for good after I did a hard reset on it on one day when I was in a hurry. I hardly ever used that computer, almost never, and there it broke on me. It had been to the shop previously and the smoozer over there didn't even bother to upgrade the 1 GB of RAM, but he did come over and sit on his ass and chatter with not me for well over an hour. I was told it was just so much better after that. Not. Wrong. Nonsense. Turned out to be a power supply issue. It was one of those really, really cheap power supplies. I opened the box to disconnect the hard drive before evaluating the hardware. Couldn't get the optical drive to load a liveCD, really weird activity. Swapped it with a quality power supply and that did it.


Thanks all.

Be real, be sober.

[WSmart]

One alternative to that crappy flash memory stuff is a RAM drive. Gigabyte has a bastard product, the I-RAM, which does work in Linux after running the setup in Wine and creating a partition table. It's only 4 GB, but mounting a RAM drive partition in place of offending directories would likely improve the situation. Of course if for some reason the I-RAM battery fails and you completely disconnect the power supply, you'd lose whatever you had on it.

Oh and I wanted to say, interesting problem, thanks for posting.

[WSmart]

I specifically mentioned backing up /home. The whole procedure of backup, installing a new distro, and then restoring the previous one takes about 1 hour.

I've been wondering about that. I'll have to try it just to try it. Simple but critical skill says I. Thanks for posting.

[Vadi]

Ahh thanks allquixotic, that makes a fair bit of sense all in all. I'll try your ideas.

I'm a bit unsure about memory, compiz is only using 50mb right now and 800mb is used in total - though perhaps if more was available, it'd cache more as well.

This is a laptop, so power supply isn't really an issue.

[WSmart]

How much hard drive swap for RAM are you actually using? Could you post output from swapon -sv when your experiencing lag. Thanks.

swapon -sv

[Vadi]

It's really not swapping, it's not out of memory when it's being so slow.

Filename				Type		Size	Used	Priority
/dev/mapper/cryptswap1                  partition	1820668	29016	-1

[WSmart]

Look at the amount of RAM you have allocated for graphics swap in your system bios. Not sure if that would go to hard drive if you're over the allocation even with RAM available. Maybe the desktop effects is too much and the graphics are using swap on the hard drive. Just a thought.

[Vadi]

I've looked in there, but it doesn't seem I can change much options at all in regards to that.

[WSmart]

Best place to look for details is a PDF manual if you can find one. Buy a $30 dollar name brand ATX board off ebay and you likely can find a PDF that details every switch, jumper and other features including the bios, but a pricey laptop....maybe not..

Pretty interesting though. Keep us posted.