Computer is lagging when there is high I/O on ext. HDD
Computer is lagging when there is high I/O on ext. HDD
Hello,
i seen this a few times for last week of my usage of MX Linux, that when there was high activity on my external HDD, whole system was lagging (including delays like 3 seconds in reaction of system menus like MX menu).
Above image probably reflect this activity that is going on on that external drive:
- checking large amount of data using torrent client on that ext. drive
- around 4MB/s download to encrypted container on that ext. drive
CPU utilization like 20%
Is this normal activity that the work space/apps reaction time is significantly badly affected even CPU, RAM and system drive is not fully utilized?
If i need to execute any specific commands at a time of lagging, let me know, or i apologize if this is bad section for this topic.
i seen this a few times for last week of my usage of MX Linux, that when there was high activity on my external HDD, whole system was lagging (including delays like 3 seconds in reaction of system menus like MX menu).
Above image probably reflect this activity that is going on on that external drive:
- checking large amount of data using torrent client on that ext. drive
- around 4MB/s download to encrypted container on that ext. drive
CPU utilization like 20%
Is this normal activity that the work space/apps reaction time is significantly badly affected even CPU, RAM and system drive is not fully utilized?
If i need to execute any specific commands at a time of lagging, let me know, or i apologize if this is bad section for this topic.
OS: MXLinux 17, systemd; GPU: AMD Radeon RX 560
Re: Computer is lagging when there is high I/O on ext. HDD
That's my experience too. Bad I/O can bring the system down.
E.g. plugging in a bad USB stick, can slow down the whole system significantly.
Gigabyte Z77M-D3H, Intel Xeon E3-1240 V2 (Quad core), 32GB RAM,
GeForce GTX 770, Samsung SSD 850 EVO 500GB, Seagate Barracuda 4TB
GeForce GTX 770, Samsung SSD 850 EVO 500GB, Seagate Barracuda 4TB
Re: Computer is lagging when there is high I/O on ext. HDD
It's normal to some extent, I think you can improve this if you know a program causes it by lowering the IO priority with ionice. https://linux.die.net/man/1/ionice
Re: Computer is lagging when there is high I/O on ext. HDD
Copying $BIG_FILES to/from USB devices is one of the few areas in MX where I sometimes wish I was back in Windows (I occasionally do boot into Windows just to copy stuff between two USB sticks (those are especially bad) because MX is just so slow).
What helps sometimes but not always is using the nocache utility. This avoids contaminating the i/o cache with stuff that presumably doesn't need to be cached in the first place. See this post: viewtopic.php?f=104&t=46213#p460366 for details.
YMMV.
What helps sometimes but not always is using the nocache utility. This avoids contaminating the i/o cache with stuff that presumably doesn't need to be cached in the first place. See this post: viewtopic.php?f=104&t=46213#p460366 for details.
YMMV.
Frugal installs on Lenovo ThinkPad L14 Ryzen 5 4650U/24GB * HP Pavilion Ryzen 3 3300U/16GB * Toshiba R950 i5-3340M/12GB
I have a reservation... What do you mean it's not in the COMPUTER!
I have a reservation... What do you mean it's not in the COMPUTER!
-
- Posts: 136
- Joined: Sat May 02, 2015 4:35 pm
Re: Computer is lagging when there is high I/O on ext. HDD
NTFS and exFat? Are you stuck with those?
USB is a CPU mastered buss, so it makes sense that you'd notice a slow down if there was high contention on a USB filesystem especially an encrypted one.
I don't dual boot or have to deal with Windows much, so I got rid of NTFS and exFat after I discovered how much they were bogging down my processes.
I run 5 I/O tests on each of 8 filesystems whenever I think about upgrading to a kernel. NTFS is consistently the least performant FS on MX17. And write speeds are especially poor.
This isn't the fault of MX. NTFS and exFat are patented, proprietary filesystems and only Microsoft knows how they really work. Linux implementations are reverse-engineered.
If you can move to a more performant file system, the problem may just vanish. Here are two quick charts from my test bed DB.
Here, NTFS (light blue bars) is dead last in overall IOPS/sec running in all the kernels. I should mention that JFS is the least CPU intensive, but it tends to fragment a bit more than the other Linux FSs. On this second chart, I've highlighted the NTFS write speeds in red compared to EXT4 and F2FS.
USB is a CPU mastered buss, so it makes sense that you'd notice a slow down if there was high contention on a USB filesystem especially an encrypted one.
I don't dual boot or have to deal with Windows much, so I got rid of NTFS and exFat after I discovered how much they were bogging down my processes.
I run 5 I/O tests on each of 8 filesystems whenever I think about upgrading to a kernel. NTFS is consistently the least performant FS on MX17. And write speeds are especially poor.
This isn't the fault of MX. NTFS and exFat are patented, proprietary filesystems and only Microsoft knows how they really work. Linux implementations are reverse-engineered.
If you can move to a more performant file system, the problem may just vanish. Here are two quick charts from my test bed DB.
Here, NTFS (light blue bars) is dead last in overall IOPS/sec running in all the kernels. I should mention that JFS is the least CPU intensive, but it tends to fragment a bit more than the other Linux FSs. On this second chart, I've highlighted the NTFS write speeds in red compared to EXT4 and F2FS.
You do not have the required permissions to view the files attached to this post.
Son, someday all this will belong to your ex wife.
Re: Computer is lagging when there is high I/O on ext. HDD
Yes, indeed. Mostly NTFS, very rarely FAT32.
Hm... perhaps I should, just for the sake of it, format two of unused 32GB sticks as ext4 and simply see what happens when I copy ~32GB between them.clicktician wrote: ↑Wed Nov 07, 2018 1:07 pmI don't dual boot or have to deal with Windows much, so I got rid of NTFS and exFat after I discovered how much they were bogging down my processes.
I run 5 I/O tests on each of 8 filesystems whenever I think about upgrading to a kernel. NTFS is consistently the least performant FS on MX17. And write speeds are especially poor.
However, while this is certainly interesting from a technical POV it doesn't solve the problem as I have to keep those bigger sticks and HDDs as NTFS for the foreseeable future. (True even if I could get rid of Windows on my PCs... one set of disks holds music and photos, another videos and these are used with my TVs and various other "smart" devices which able to read NTFS but not ext4. )
Frugal installs on Lenovo ThinkPad L14 Ryzen 5 4650U/24GB * HP Pavilion Ryzen 3 3300U/16GB * Toshiba R950 i5-3340M/12GB
I have a reservation... What do you mean it's not in the COMPUTER!
I have a reservation... What do you mean it's not in the COMPUTER!
-
- Posts: 136
- Joined: Sat May 02, 2015 4:35 pm
Re: Computer is lagging when there is high I/O on ext. HDD
Copying 32GB of files onto a 32GB stick... did you mean 64GB stick? If you really want to run your sticks at 100% full, EXT4 (or any EXT FS or XFS) is probably not the best choice.
The reason is, these filesystems leave freespace between files in anticipation of resizing them later, and it spreads the files out over the partition. This "freespace fragmentation" allows for the easily adjustable extents -- hence the name EXTX. This helps control "file fragmentation" (the second type of frag, and that's the bad kind).
As the EXTX disk becomes full, tho, the FS will attempt to squeeze new file parts into the ever-shrinking spare extent space it left open adjacent to other files, which results in terrible fragged performance when reading and writing the last bits.
As a rule, I never fill an EXT file system past 80%, because performance suffers.
If you want to wipe the stick and then fill it up 100%, I would suggest JFS instead. In my testing, it is lightest on CPU usage, it's often faster than EXT type filesystems. But because it doesn't manage extents well, it gets "file fragmentation" with repeated deletes, adds and updates. But if you're deleting everything (or formatting) and then re-writing the whole stick, you don't have to worry about any file fragmentation. You're just writing all your files back-to-back which is how JFS should store them.
Son, someday all this will belong to your ex wife.
Re: Computer is lagging when there is high I/O on ext. HDD
@clicktician: I was/am thinking to do such a ~32GB copying exercise simply as a test as to whether ext4 on two USB sticks is indeed faster than the same setup but with NTFS. This is not and will never be a production issue since I have to stick with NTFS as explained earlier. However, it would certainly be interesting to see whether ext4 is indeed faster than NTFS in that scenario, so I will try it.
Frugal installs on Lenovo ThinkPad L14 Ryzen 5 4650U/24GB * HP Pavilion Ryzen 3 3300U/16GB * Toshiba R950 i5-3340M/12GB
I have a reservation... What do you mean it's not in the COMPUTER!
I have a reservation... What do you mean it's not in the COMPUTER!
Re: Computer is lagging when there is high I/O on ext. HDD
Sorry to bother you all. Apologize for my ignorance, but i really do not know how to discover root cause and fix it. I believe that high utilization of the external USB 2.0 HDD (respectively the virtual HDD - decrypted Veracrypt container) is not a reason to the MX Linux to be lagging while my system HDD, CPU and RAM has some room. I think this is problem in Linux. Linux should be able to do its job, no matter what slow external HDD does? The ext. HDD is only the bottleneck of the process that is using it. Why should MX menu (responsiveness) and Firefox responsiveness etc be affected by it?
OS: MXLinux 17, systemd; GPU: AMD Radeon RX 560
-
- Posts: 136
- Joined: Sat May 02, 2015 4:35 pm
Re: Computer is lagging when there is high I/O on ext. HDD
Are your mouse and keyboard also USB devices? <I don't know> If so, they are sharing the same buss as your busy disk device.
That would every application interface that uses your mouse and keyboard appear to respond slowly to your human actions.
Son, someday all this will belong to your ex wife.