USB transfer speeds slow/ data needs to be written message

[ronjnk]

I have a weird problem with USB thumb drives. This proiblem occurs regardless of which thumb drive I utilize. This is a relatively new MX 18 install within the last couple weeks and this is my first attempt to write to them since installation. I had no problem transferring data that was on the USB drives to my computer but writing is brutally slow. I should mention that I have had no problem reading and writing to my WD 500Gig hard drive. That's fast both ways. Read/write.

It took about 5 minutes to transfer 1 gig of data to my thumb drive. It is formatted FAT32. So I reformatted to NTFS. Transfer speed increased for the 1 gig transfer to under 15 seconds. Then I unmount and it says "There is data that needs to be written to the device before it can be removed. That process takes 5 minutes or more to complete. I have no idea what it is doing but the black message box associated with it sits prominently on the screen blocking some of the screen until it completes the unmount.

I tried to transfer 7 gigs of data to the same USB drive. It starts out transferring fast but after a few seconds, I can see it start to slow down and the calculated minutes rise. It starts at over 125 MB/s and quickly drops to 12 MB/s. The first gig transfers fast and then takes roughly 5 minutes to transfer the last 6 gigs. Then I try to unmount but it sits there for many minutes with the "writing data to the device" message. I've never seen this problem on any windows or linux I've used before. But this is a new computer. I suspect it is a settings or formatting issue. Format is msdos and NTFS or FAT32. I've used my windows laptop to reformat as well as Gparted. I've used different USB devices and different USB slots on the computer. The behavior is the same.

Why does data transfer fast initially and then bog down to turtle speed? Fat 32 is even slower which is why I went with NTFS format. And why when I try to unmount is it writing data for a long time? Is it related to this bug? https://bugs.launchpad.net/archlinux/+s ... ug/1762595 Any idea how to fix this? Thanks! Ron

[BitJam]

Most of these problems are related to a minor bug in Linux. The default buffer size is gianormous. So writing at first is fast because it is mostly writing to RAM until the buffer fills. Then it takes a long time to do the rest of the writing because after the buffer is full, the speed is limited by the speed of the device.

I always run the "sync" command before I unmount and remove a usb stick. This makes sure all the writing gets done to the actual device.

To fix the buffer problem, I suggest you run the command:

Code: Select all

sudo sysctl vm.dirty_bytes=20000000

You could also add this to your /etc/rc.local script without the "sudo".

Our live-usb-maker programs do this and it actually increases the transfer rate by about 10%. We originally did it so the progress bar wasn't fooled by the huge Linux buffer. Using the default buffer size, the bar would say the whole thing was done in 10 or 20 seconds then you would have to wait about a minute for it to actually finish.

If you reduce the Linux buffer size then the writing should happen at a more constant speed and you will seldom have that big wait at the end for the writing to complete. You can usually just wait a second after the write command is done and you are good to go without running "sync".

There is a related problem with some usb sticks. They can have their own internal RAM buffer. So the first writes are very fast because they go to RAM but then when that RAM buffer gets full, the writing is limited by the speed of the flash memory which is much slower. Often these usb sticks with built in RAM buffers post really great specs even though their flash memory is slow.

I have to system monitors running (conky and gkrellm) and they each show me the disk activity in a rolling graph. That helps me understand what is going on. For usb sticks with RAM buffers you can see the early fast writing and then see it slow down. When it suddenly drops to near zero (assuming nothing else big is going on) then it is safe to unmount the drive.

HTH

[timkb4cq]

A lot does depend on the flash drive. Most write far slower than they read While there are USB-3.0 flash drives that will write at over 200MB/s if you just go to the store and pick up a cheap Sandisk or Kingston you'll often get one that will only write at 4 - 10 MB/s with sequential writing like transferring an ISO being faster than writing a bunch of files.
4MB/s = 240MB/min so over 4 minutes to transfer 1GB and half an hour for 7GB.

[Eadwine Rose]

I have seen this a few times with my sdcard.. basically I need to close all windows that had anything to do with the transfer before doing the unmount and then things happen properly and normally.

[ronjnk]

Good Morning Folks,

Thank you very much for the feedback.

BitJam...Right on. Your explanation of what is happening makes a great deal of sense. I was so baffled as to the initial speed and then slowing down. It was only filling up a buffer for the most part. And then when trying to unmount, I received that lengthy message that it needed to write more data and that was simply the system still writing data to the USB drive. So my assessment and worry that I had something messed up was wrong.

The suggested command you gave me worked like a charm.

Code: Select all

sudo sysctl vm.dirty_bytes=20000000

I ran your command and then transferred 7.7 gigs to USB, when finished, I hit unmount and it was immediately unmounted. Done deal. No message.

I have a couple of questions. Is that command permanent now in my system? In other words, I don't need to open terminal and input that code every time I want to make a large file transfer to USB? I did not use and am not familiar with the sync command. I don't think it is necessary in my case since the behavior seems great now, but for future reference and to perhaps help others that browse this thread, could you explain further how to use the sync command and what it is actually doing? Also, you wrote

You could also add this to your /etc/rc.local script without the "sudo".

I'm uncertain how to properly do that and I'm wondering if I really even need to do that unless the above command is only temporary and disappears once I restart the computer. Thank you BitJam!

timkb4cq... You were right! The speeds I saw were typical. I proved it by transferring the same data from a windows computer and the speeds were the same. What really threw me off was the initial speed which was only filling a buffer ( I had no idea) and then the "writing data" message which lasted so long. Although I tried different USB devices and drives, they are all usb 2.0. Thanks for your input!

Eadwine Rose...I actually thought I had another problem. When I saved to USB, unmounted and then reinserted my USB, many times when I right clicked on the drive, various commands like rename, copy, delete were grayed out and unavailable. Then I realized I had several Thunar file system windows open. It didn't like that. As soon as I closed any open file system windows, all the commands were available. So good advice as a general rule to close any file system windows. Thanks for your input!

Have a great day! Ron

[crazysquirrel]

After reading the fixes, I wonder why MX doesn't have them preset on initial install.

I am having slow speeds too.
WD Black External 5TB USB 3/0 connected to ASUS x401A USB port with included cable. Laptop has a 2TB hybrid hard drive, 8GB Ram.

Transfer speed started out at a paltry 60MB/S then dropped to 50 and I expect it to drop to near zero by morning like it did a few days ago.
60MB/S is like 1/10th the speed it should be getting.

I tried all this a few days ago and suddenly the computer wouldn't dual boot back to win7 (Dual booting Win7 and MX 19.2 X64) And MX took 5 min to boot.

Had to R/R the whole drive and start all over without getting any updates for either.
I pay by the GB for data and cannot afford to experiment.
I installed win7 first and did some basic scan disking.
No problems found so far.
I did not do a thorough (surface) scan disk as that would take days to do.

When reinstalling MX 19.2, I did check the box for checking for bad blocks. None found.

Oh and cpu temps SOARED to an average 150F !

Only thing plugged in is the external drive and have the screen turned off to reduce heat.

Side note: ASUS sent me an email stating that the cpu on all their laptops are soldered onto the motherboard.

I also discovered that most external hard drives now are not 'harvestable' due to the USB connector soldered onto the hard drive.

Seems ANYTHING to force you to spend more money.

I would need step by step EXACT steps on what to do (assume nothing please) on how to change my files to speed things up.
I do not have a swap file - had one the first time and none this time.

Exact meaning:
Left click on (whatever)
Scroll till you see (whatever) and left click on that.
If I need to choose what to open something with I need that info too.
What to type and where to put it.
Step by step. Help manual does not have sufficient information. Like to NEVER find the screensaver...

For me, MX is like visiting a foreign country (USA vs UK for example - similar but not the same).

[JayM]

According to the specs the Asus x401a has a USB2.0 port and a USB3.0 port. Make sure you're using the right one.

Please start new threads for your other issues and include your Quick System Info.

[crazysquirrel]

Yes I am. The USB 2 port is where you plug in external power. USB port opposite corner and blue. External plugged into the 3.0 port.

Oh and the transfer is transferring FROM the external drive.
Similar experience transferring to the drive last week.

Program reports a whopping 5 hours to transfer. By morning I expect it to say 1822 hours with 2/3 more to go yet like it did last time.
No kidding either.
It takes a lot longer than what is reported.

[JayM]

Sounds like you either have a bad laptop or a bad external drive or USB cable. There's not much that MX Linux can do about hardware issues. And if your system is getting as hot as you say it is that's probably the cause of the slowdown due to thermal throttling of the CPU. Is the fan working? It may be clogged with dust. Search the web for info about how to clean it.

Edit: 150F is only 65C which isn't all that hot and is within normal range for most CPUs. I don't know about yours as you haven't supplied your Quick System Info when asking for help (as per the forum rules) so I don't know what your CPU is.

[crazysquirrel]

HD and cable are brand new (less than 2 weeks old).
I do feel heat coming out of the side.
We did clean it about a year ago and even then it was not dirty.
I will have to wait till tomorrow when it is done to take it apart and check.
PS it is now down to 17MB/S and dropping.
200GB transferred out of 1.5 TB.