Computer is lagging when there is high I/O on ext. HDD

[postcd]

Hello,

i seen this a few times for last week of my usage of MX Linux, that when there was high activity on my external HDD, whole system was lagging (including delays like 3 seconds in reaction of system menus like MX menu).


Above image probably reflect this activity that is going on on that external drive:
- checking large amount of data using torrent client on that ext. drive
- around 4MB/s download to encrypted container on that ext. drive
CPU utilization like 20%

Is this normal activity that the work space/apps reaction time is significantly badly affected even CPU, RAM and system drive is not fully utilized?

If i need to execute any specific commands at a time of lagging, let me know, or i apologize if this is bad section for this topic.

[fehlix]

Is this normal activity that the work space/apps reaction time is significantly badly affected even CPU, RAM and system drive is not fully utilized?

That's my experience too. Bad I/O can bring the system down.
E.g. plugging in a bad USB stick, can slow down the whole system significantly.

[Adrian]

It's normal to some extent, I think you can improve this if you know a program causes it by lowering the IO priority with ionice. https://linux.die.net/man/1/ionice

[thomasl]

Copying $BIG_FILES to/from USB devices is one of the few areas in MX where I sometimes wish I was back in Windows (I occasionally do boot into Windows just to copy stuff between two USB sticks (those are especially bad) because MX is just so slow).

What helps sometimes but not always is using the nocache utility. This avoids contaminating the i/o cache with stuff that presumably doesn't need to be cached in the first place. See this post: viewtopic.php?f=104&t=46213#p460366 for details.

YMMV.

[clicktician]

NTFS and exFat? Are you stuck with those?
USB is a CPU mastered buss, so it makes sense that you'd notice a slow down if there was high contention on a USB filesystem especially an encrypted one.

I don't dual boot or have to deal with Windows much, so I got rid of NTFS and exFat after I discovered how much they were bogging down my processes.
I run 5 I/O tests on each of 8 filesystems whenever I think about upgrading to a kernel. NTFS is consistently the least performant FS on MX17. And write speeds are especially poor.
This isn't the fault of MX. NTFS and exFat are patented, proprietary filesystems and only Microsoft knows how they really work. Linux implementations are reverse-engineered.

If you can move to a more performant file system, the problem may just vanish. Here are two quick charts from my test bed DB.
Here, NTFS (light blue bars) is dead last in overall IOPS/sec running in all the kernels. I should mention that JFS is the least CPU intensive, but it tends to fragment a bit more than the other Linux FSs.

fs_speed_chart.jpg

On this second chart, I've highlighted the NTFS write speeds in red compared to EXT4 and F2FS.

ext4_f2fs_ntfs_detail.jpg

[thomasl]

NTFS and exFat? Are you stuck with those?

Yes, indeed. Mostly NTFS, very rarely FAT32.

I don't dual boot or have to deal with Windows much, so I got rid of NTFS and exFat after I discovered how much they were bogging down my processes.
I run 5 I/O tests on each of 8 filesystems whenever I think about upgrading to a kernel. NTFS is consistently the least performant FS on MX17. And write speeds are especially poor.

Hm... perhaps I should, just for the sake of it, format two of unused 32GB sticks as ext4 and simply see what happens when I copy ~32GB between them.

However, while this is certainly interesting from a technical POV it doesn't solve the problem as I have to keep those bigger sticks and HDDs as NTFS for the foreseeable future. (True even if I could get rid of Windows on my PCs... one set of disks holds music and photos, another videos and these are used with my TVs and various other "smart" devices which able to read NTFS but not ext4. )

[clicktician]

perhaps I should, just for the sake of it, format two of unused 32GB sticks as ext4 and simply see what happens when I copy ~32GB between them.

Copying 32GB of files onto a 32GB stick... did you mean 64GB stick? If you really want to run your sticks at 100% full, EXT4 (or any EXT FS or XFS) is probably not the best choice.
The reason is, these filesystems leave freespace between files in anticipation of resizing them later, and it spreads the files out over the partition. This "freespace fragmentation" allows for the easily adjustable extents -- hence the name EXTX. This helps control "file fragmentation" (the second type of frag, and that's the bad kind).

As the EXTX disk becomes full, tho, the FS will attempt to squeeze new file parts into the ever-shrinking spare extent space it left open adjacent to other files, which results in terrible fragged performance when reading and writing the last bits.
As a rule, I never fill an EXT file system past 80%, because performance suffers.

If you want to wipe the stick and then fill it up 100%, I would suggest JFS instead. In my testing, it is lightest on CPU usage, it's often faster than EXT type filesystems. But because it doesn't manage extents well, it gets "file fragmentation" with repeated deletes, adds and updates. But if you're deleting everything (or formatting) and then re-writing the whole stick, you don't have to worry about any file fragmentation. You're just writing all your files back-to-back which is how JFS should store them.

[thomasl]

@clicktician: I was/am thinking to do such a ~32GB copying exercise simply as a test as to whether ext4 on two USB sticks is indeed faster than the same setup but with NTFS. This is not and will never be a production issue since I have to stick with NTFS as explained earlier. However, it would certainly be interesting to see whether ext4 is indeed faster than NTFS in that scenario, so I will try it.

[postcd]

Sorry to bother you all. Apologize for my ignorance, but i really do not know how to discover root cause and fix it. I believe that high utilization of the external USB 2.0 HDD (respectively the virtual HDD - decrypted Veracrypt container) is not a reason to the MX Linux to be lagging while my system HDD, CPU and RAM has some room. I think this is problem in Linux. Linux should be able to do its job, no matter what slow external HDD does? The ext. HDD is only the bottleneck of the process that is using it. Why should MX menu (responsiveness) and Firefox responsiveness etc be affected by it?

[clicktician]

Why should MX menu (responsiveness) and Firefox responsiveness etc be affected by it?

Are your mouse and keyboard also USB devices? <I don't know> If so, they are sharing the same buss as your busy disk device.
That would every application interface that uses your mouse and keyboard appear to respond slowly to your human actions.