Welcome!

The kernel problem with recent updates has been solved. Find the solution here

Important information
-- Required MX 15/16 Repository Changes
-- Information on torrent hosting changes
-- Information on MX15/16 GPG Keys
-- Spectre and Meltdown vulnerabilities

News
-- Introducing our new Website
-- MX Linux on social media: here

Current releases
-- MX-18.3 Point Release release info here
-- Migration Information to MX-18 here
-- antiX-17.4.1 release info here

New users
-- Please read this first, and don't forget to add system and hardware information to posts!
-- Here are the Forum Rules

USB transfer speeds slow/ data needs to be written message

Post Reply
ronjnk
Forum Novice
Forum  Novice
Posts: 40
Joined: Wed Dec 19, 2018 9:36 pm

USB transfer speeds slow/ data needs to be written message

#1

Post by ronjnk » Thu Jan 03, 2019 9:27 pm

I have a weird problem with USB thumb drives. This proiblem occurs regardless of which thumb drive I utilize. This is a relatively new MX 18 install within the last couple weeks and this is my first attempt to write to them since installation. I had no problem transferring data that was on the USB drives to my computer but writing is brutally slow. I should mention that I have had no problem reading and writing to my WD 500Gig hard drive. That's fast both ways. Read/write.

It took about 5 minutes to transfer 1 gig of data to my thumb drive. It is formatted FAT32. So I reformatted to NTFS. Transfer speed increased for the 1 gig transfer to under 15 seconds. Then I unmount and it says "There is data that needs to be written to the device before it can be removed. That process takes 5 minutes or more to complete. I have no idea what it is doing but the black message box associated with it sits prominently on the screen blocking some of the screen until it completes the unmount.

I tried to transfer 7 gigs of data to the same USB drive. It starts out transferring fast but after a few seconds, I can see it start to slow down and the calculated minutes rise. It starts at over 125 MB/s and quickly drops to 12 MB/s. The first gig transfers fast and then takes roughly 5 minutes to transfer the last 6 gigs. Then I try to unmount but it sits there for many minutes with the "writing data to the device" message. I've never seen this problem on any windows or linux I've used before. But this is a new computer. I suspect it is a settings or formatting issue. Format is msdos and NTFS or FAT32. I've used my windows laptop to reformat as well as Gparted. I've used different USB devices and different USB slots on the computer. The behavior is the same.

Why does data transfer fast initially and then bog down to turtle speed? Fat 32 is even slower which is why I went with NTFS format. And why when I try to unmount is it writing data for a long time? Is it related to this bug? https://bugs.launchpad.net/archlinux/+s ... ug/1762595 Any idea how to fix this? Thanks! Ron

User avatar
BitJam
Forum Veteran
Forum Veteran
Posts: 3347
Joined: Sat Aug 22, 2009 11:36 pm

Re: USB transfer speeds slow/ data needs to be written message

#2

Post by BitJam » Thu Jan 03, 2019 10:16 pm

Most of these problems are related to a minor bug in Linux. The default buffer size is gianormous. So writing at first is fast because it is mostly writing to RAM until the buffer fills. Then it takes a long time to do the rest of the writing because after the buffer is full, the speed is limited by the speed of the device.

I always run the "sync" command before I unmount and remove a usb stick. This makes sure all the writing gets done to the actual device.

To fix the buffer problem, I suggest you run the command:

Code: Select all

sudo sysctl vm.dirty_bytes=20000000
You could also add this to your /etc/rc.local script without the "sudo".

Our live-usb-maker programs do this and it actually increases the transfer rate by about 10%. We originally did it so the progress bar wasn't fooled by the huge Linux buffer. Using the default buffer size, the bar would say the whole thing was done in 10 or 20 seconds then you would have to wait about a minute for it to actually finish.

If you reduce the Linux buffer size then the writing should happen at a more constant speed and you will seldom have that big wait at the end for the writing to complete. You can usually just wait a second after the write command is done and you are good to go without running "sync".

There is a related problem with some usb sticks. They can have their own internal RAM buffer. So the first writes are very fast because they go to RAM but then when that RAM buffer gets full, the writing is limited by the speed of the flash memory which is much slower. Often these usb sticks with built in RAM buffers post really great specs even though their flash memory is slow.

I have to system monitors running (conky and gkrellm) and they each show me the disk activity in a rolling graph. That helps me understand what is going on. For usb sticks with RAM buffers you can see the early fast writing and then see it slow down. When it suddenly drops to near zero (assuming nothing else big is going on) then it is safe to unmount the drive.

HTH
Will I cry when it's all over?
When I die will I see Heaven?

User avatar
timkb4cq
Forum Veteran
Forum Veteran
Posts: 5121
Joined: Wed Jul 12, 2006 4:05 pm

Re: USB transfer speeds slow/ data needs to be written message

#3

Post by timkb4cq » Thu Jan 03, 2019 10:29 pm

A lot does depend on the flash drive. Most write far slower than they read While there are USB-3.0 flash drives that will write at over 200MB/s if you just go to the store and pick up a cheap Sandisk or Kingston you'll often get one that will only write at 4 - 10 MB/s with sequential writing like transferring an ISO being faster than writing a bunch of files.
4MB/s = 240MB/min so over 4 minutes to transfer 1GB and half an hour for 7GB.
MSI 970A-G43 MB, AMD FX-6300 (six core), 16GB RAM, GeForce 730, Samsung 850 EVO 250GB SSD, Seagate Barracuda XT 3TB

User avatar
Eadwine Rose
Forum Veteran
Forum Veteran
Posts: 8946
Joined: Wed Jul 12, 2006 2:10 am

Re: USB transfer speeds slow/ data needs to be written message

#4

Post by Eadwine Rose » Thu Jan 03, 2019 11:37 pm

I have seen this a few times with my sdcard.. basically I need to close all windows that had anything to do with the transfer before doing the unmount and then things happen properly and normally.
MX-18.3_x64 May 26 2019 * 4.19.0-5-amd64 ext4 Xfce 4.12.3 * 8-core AMD Ryzen 7 2700
Asus TUF B450-Plus Gaming UEFI * Asus GTX 1050 Ti Nvidia 390.116 * 2x16Gb DDR4 2666 Kingston HyperX Predator
Samsung 860EVO * Samsung S24D330 & P2250 * HP Envy5030

ronjnk
Forum Novice
Forum  Novice
Posts: 40
Joined: Wed Dec 19, 2018 9:36 pm

Re: USB transfer speeds slow/ data needs to be written message

#5

Post by ronjnk » Fri Jan 04, 2019 9:35 am

Good Morning Folks,

Thank you very much for the feedback.

BitJam...Right on. Your explanation of what is happening makes a great deal of sense. I was so baffled as to the initial speed and then slowing down. It was only filling up a buffer for the most part. And then when trying to unmount, I received that lengthy message that it needed to write more data and that was simply the system still writing data to the USB drive. So my assessment and worry that I had something messed up was wrong.

The suggested command you gave me worked like a charm.

Code: Select all

sudo sysctl vm.dirty_bytes=20000000
I ran your command and then transferred 7.7 gigs to USB, when finished, I hit unmount and it was immediately unmounted. Done deal. No message.

I have a couple of questions. Is that command permanent now in my system? In other words, I don't need to open terminal and input that code every time I want to make a large file transfer to USB? I did not use and am not familiar with the sync command. I don't think it is necessary in my case since the behavior seems great now, but for future reference and to perhaps help others that browse this thread, could you explain further how to use the sync command and what it is actually doing? Also, you wrote
You could also add this to your /etc/rc.local script without the "sudo".
I'm uncertain how to properly do that and I'm wondering if I really even need to do that unless the above command is only temporary and disappears once I restart the computer. Thank you BitJam!

timkb4cq... You were right! The speeds I saw were typical. I proved it by transferring the same data from a windows computer and the speeds were the same. What really threw me off was the initial speed which was only filling a buffer ( I had no idea) and then the "writing data" message which lasted so long. Although I tried different USB devices and drives, they are all usb 2.0. Thanks for your input!

Eadwine Rose...I actually thought I had another problem. When I saved to USB, unmounted and then reinserted my USB, many times when I right clicked on the drive, various commands like rename, copy, delete were grayed out and unavailable. Then I realized I had several Thunar file system windows open. It didn't like that. As soon as I closed any open file system windows, all the commands were available. So good advice as a general rule to close any file system windows. Thanks for your input!

Have a great day! Ron

Post Reply

Return to “Hardware /Configuration”