Page 1 of 1

USB transfer speeds slow/ data needs to be written message

Posted: Thu Jan 03, 2019 9:27 pm
by ronjnk
I have a weird problem with USB thumb drives. This proiblem occurs regardless of which thumb drive I utilize. This is a relatively new MX 18 install within the last couple weeks and this is my first attempt to write to them since installation. I had no problem transferring data that was on the USB drives to my computer but writing is brutally slow. I should mention that I have had no problem reading and writing to my WD 500Gig hard drive. That's fast both ways. Read/write.

It took about 5 minutes to transfer 1 gig of data to my thumb drive. It is formatted FAT32. So I reformatted to NTFS. Transfer speed increased for the 1 gig transfer to under 15 seconds. Then I unmount and it says "There is data that needs to be written to the device before it can be removed. That process takes 5 minutes or more to complete. I have no idea what it is doing but the black message box associated with it sits prominently on the screen blocking some of the screen until it completes the unmount.

I tried to transfer 7 gigs of data to the same USB drive. It starts out transferring fast but after a few seconds, I can see it start to slow down and the calculated minutes rise. It starts at over 125 MB/s and quickly drops to 12 MB/s. The first gig transfers fast and then takes roughly 5 minutes to transfer the last 6 gigs. Then I try to unmount but it sits there for many minutes with the "writing data to the device" message. I've never seen this problem on any windows or linux I've used before. But this is a new computer. I suspect it is a settings or formatting issue. Format is msdos and NTFS or FAT32. I've used my windows laptop to reformat as well as Gparted. I've used different USB devices and different USB slots on the computer. The behavior is the same.

Why does data transfer fast initially and then bog down to turtle speed? Fat 32 is even slower which is why I went with NTFS format. And why when I try to unmount is it writing data for a long time? Is it related to this bug? https://bugs.launchpad.net/archlinux/+s ... ug/1762595 Any idea how to fix this? Thanks! Ron

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Jan 03, 2019 10:16 pm
by BitJam
Most of these problems are related to a minor bug in Linux. The default buffer size is gianormous. So writing at first is fast because it is mostly writing to RAM until the buffer fills. Then it takes a long time to do the rest of the writing because after the buffer is full, the speed is limited by the speed of the device.

I always run the "sync" command before I unmount and remove a usb stick. This makes sure all the writing gets done to the actual device.

To fix the buffer problem, I suggest you run the command:

Code: Select all

sudo sysctl vm.dirty_bytes=20000000
You could also add this to your /etc/rc.local script without the "sudo".

Our live-usb-maker programs do this and it actually increases the transfer rate by about 10%. We originally did it so the progress bar wasn't fooled by the huge Linux buffer. Using the default buffer size, the bar would say the whole thing was done in 10 or 20 seconds then you would have to wait about a minute for it to actually finish.

If you reduce the Linux buffer size then the writing should happen at a more constant speed and you will seldom have that big wait at the end for the writing to complete. You can usually just wait a second after the write command is done and you are good to go without running "sync".

There is a related problem with some usb sticks. They can have their own internal RAM buffer. So the first writes are very fast because they go to RAM but then when that RAM buffer gets full, the writing is limited by the speed of the flash memory which is much slower. Often these usb sticks with built in RAM buffers post really great specs even though their flash memory is slow.

I have to system monitors running (conky and gkrellm) and they each show me the disk activity in a rolling graph. That helps me understand what is going on. For usb sticks with RAM buffers you can see the early fast writing and then see it slow down. When it suddenly drops to near zero (assuming nothing else big is going on) then it is safe to unmount the drive.

HTH

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Jan 03, 2019 10:29 pm
by timkb4cq
A lot does depend on the flash drive. Most write far slower than they read While there are USB-3.0 flash drives that will write at over 200MB/s if you just go to the store and pick up a cheap Sandisk or Kingston you'll often get one that will only write at 4 - 10 MB/s with sequential writing like transferring an ISO being faster than writing a bunch of files.
4MB/s = 240MB/min so over 4 minutes to transfer 1GB and half an hour for 7GB.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Thu Jan 03, 2019 11:37 pm
by Eadwine Rose
I have seen this a few times with my sdcard.. basically I need to close all windows that had anything to do with the transfer before doing the unmount and then things happen properly and normally.

Re: USB transfer speeds slow/ data needs to be written message

Posted: Fri Jan 04, 2019 9:35 am
by ronjnk
Good Morning Folks,

Thank you very much for the feedback.

BitJam...Right on. Your explanation of what is happening makes a great deal of sense. I was so baffled as to the initial speed and then slowing down. It was only filling up a buffer for the most part. And then when trying to unmount, I received that lengthy message that it needed to write more data and that was simply the system still writing data to the USB drive. So my assessment and worry that I had something messed up was wrong.

The suggested command you gave me worked like a charm.

Code: Select all

sudo sysctl vm.dirty_bytes=20000000
I ran your command and then transferred 7.7 gigs to USB, when finished, I hit unmount and it was immediately unmounted. Done deal. No message.

I have a couple of questions. Is that command permanent now in my system? In other words, I don't need to open terminal and input that code every time I want to make a large file transfer to USB? I did not use and am not familiar with the sync command. I don't think it is necessary in my case since the behavior seems great now, but for future reference and to perhaps help others that browse this thread, could you explain further how to use the sync command and what it is actually doing? Also, you wrote
You could also add this to your /etc/rc.local script without the "sudo".
I'm uncertain how to properly do that and I'm wondering if I really even need to do that unless the above command is only temporary and disappears once I restart the computer. Thank you BitJam!

timkb4cq... You were right! The speeds I saw were typical. I proved it by transferring the same data from a windows computer and the speeds were the same. What really threw me off was the initial speed which was only filling a buffer ( I had no idea) and then the "writing data" message which lasted so long. Although I tried different USB devices and drives, they are all usb 2.0. Thanks for your input!

Eadwine Rose...I actually thought I had another problem. When I saved to USB, unmounted and then reinserted my USB, many times when I right clicked on the drive, various commands like rename, copy, delete were grayed out and unavailable. Then I realized I had several Thunar file system windows open. It didn't like that. As soon as I closed any open file system windows, all the commands were available. So good advice as a general rule to close any file system windows. Thanks for your input!

Have a great day! Ron