Welcome!

The kernel problem with recent updates has been solved. Find the solution here

Important information
-- Required MX 15/16 Repository Changes
-- Information on torrent hosting changes
-- Information on MX15/16 GPG Keys
-- Spectre and Meltdown vulnerabilities

News
-- Introducing our new Website
-- MX Linux on social media: here

Current releases
-- MX-18.3 Point Release release info here
-- Migration Information to MX-18 here
-- antiX-17.4.1 release info here

New users
-- Please read this first, and don't forget to add system and hardware information to posts!
-- Here are the Forum Rules

(MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

User avatar
dolphin_oracle
Forum Veteran
Forum Veteran
Posts: 12142
Joined: Sun Dec 16, 2007 1:17 pm

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#11

Post by dolphin_oracle » Tue Jul 17, 2018 2:13 pm

nutrialex wrote:
Tue Jul 17, 2018 2:09 pm
I am transferring movie files (from about 1 gb to 8gb) to hard drives and usb sticks very often. Lets say that I want to transfer 10 files of 1gb each at a total of 10gb. I said that I have the same problem regardless I am using swap or not. I tried it both.
I missed the swap thing, thanks.

fyi fat32 would not work for files over about 3.5 GB.

did you try the mount options provided by truongtfg? (flush vs. sync?)


***edit*** actually never mind on that mount option, its already "flush" by default.

you can check the mount options in use with

Code: Select all

cat /proc/mounts
http://www.youtube.com/runwiththedolphin
lenovo ThinkPad T530 - MX-18.3
lenovo s21e - antiX-19 beta 2 (live-USB)
FYI: mx "test" repo is not the same thing as debian testing repo.

truongtfg
Forum Novice
Forum  Novice
Posts: 32
Joined: Sun Jul 15, 2018 9:22 am

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#12

Post by truongtfg » Tue Jul 17, 2018 11:33 pm

Here are the solutions that I gathered from the net that you can try:

1. unmount, remove drive, and run sudo modprobe ehci_hcd in the Terminal. Insert drive and again sudo modprobe ehci_hcd

2. umount the device if it is automounted already, and manually mount it to /mnt/foldername (no idea why this works for someone)
For example,

Code: Select all

umount /media/usb0
mount /dev/sdb1 /mnt/sam
3. If the drive is already mounted with flush, unmount and mount it again with sync, if it is still not working try removing both the flush and sync (if any) in /etc/fstab

By looking for the corresponding line from

Code: Select all

cat /proc/mounts
. It may look something like:

Code: Select all

/dev/sdb1 /media/usb0 fuseblk,sync,rw,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0
See that 'sync'? Edit the /etc/fstab and remove it.

I'm at work now, when I get home I'll try formatting some old pen drive to FAT32 and see that if I can replicate your case.

Good luck.

Edit: Please post your result from cat /proc/mounts so that we can see what your mount option is

nutrialex
Forum Novice
Forum  Novice
Posts: 38
Joined: Wed Oct 21, 2015 3:40 am

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#13

Post by nutrialex » Wed Jul 18, 2018 3:02 am

Here is my mounts

Code: Select all

alex@HTPC:~
$ cat /proc/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=3034616k,nr_inodes=758654,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=610464k,mode=755 0 0
/dev/sda1 / ext4 rw,relatime 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
pstore /sys/fs/pstore pstore rw,relatime 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=1220920k 0 0
/dev/sda2 /home ext4 rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/sdb1 /home/alex/3TB fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0
/dev/sdd1 /home/alex/1 fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0
/dev/sdc1 /home/alex/2 fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0
/dev/sdf1 /home/alex/Tainies fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0
/dev/sdg1 /home/alex/White fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
cgroup /sys/fs/cgroup tmpfs rw,relatime,size=12k,mode=755 0 0
systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,release_agent=/run/cgmanager/agents/cgm-release-agent.systemd,name=systemd 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=610460k,mode=700,uid=1000,gid=1000 0 0
gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
/dev/sde1 /home/alex/5TB fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0
/dev/sdh1 /media/alex/2312-5338 vfat rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=utf8,shortname=mixed,showexec,utf8,flush,errors=remount-ro 0 0
The last line is my Fat32 usb

Code: Select all

/dev/sdh1 /media/alex/2312-5338 vfat rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=utf8,shortname=mixed,showexec,utf8,flush,errors=remount-ro 0 0
There is no fuse or sync.
I will try the rest of your ideas in a while.

User avatar
fehlix
Forum Veteran
Forum Veteran
Posts: 4580
Joined: Wed Apr 11, 2018 5:09 pm

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#14

Post by fehlix » Wed Jul 18, 2018 9:30 am

Would you mind showing us the fstab just to get a clearer picture, e.g.

Code: Select all

cat /etc/fstab
And in addition show us what inxi tells us about the usb‘s

Code: Select all

inxi -Dxx --usb
Thanks
Gigabyte Z77M-D3H, Intel Xeon E3-1240 V2 (Quad core), 32GB RAM,
GeForce GTX 770, Samsung SSD 850 EVO 500GB, Seagate Barracuda 4TB

nutrialex
Forum Novice
Forum  Novice
Posts: 38
Joined: Wed Oct 21, 2015 3:40 am

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#15

Post by nutrialex » Wed Jul 18, 2018 10:18 am

Thank you for your efforts,

my fstab:

Code: Select all

alex@HTPC:~
$ cat  /etc/fstab
# /etc/fstab: static file system information
#
# Created by make-fstab on Wed Jul 18 03:36:44 EDT 2018

# <file system>                            <mount point>                               <type>     <options>                       <dump/pass>

#-> /dev/sda1  label=MX17.1
UUID=8dbb220a-2c96-4c49-987f-beae063a41e7  /                                           ext4       defaults                        1 1
#-> /dev/sda2  label=homeMX
UUID=e8ac3b1e-b614-4294-99d0-12bf57dbea57  /home                                       ext4       defaults                        1 2

#Entry for /dev/sdb1 :
UUID=521C86F176C46EE1	/home/alex/3TB		ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	00
#Entry for /dev/sdd1 :
UUID=EE98A92E98A8F66D	/home/alex/1		ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	00
#Entry for /dev/sdc1 :
UUID=4AE86E39E86E2383	/home/alex/2		ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	00
#Entry for /dev/sde1 :
UUID=656438C1331FFE55	/home/alex/5TB		ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	00
#Entry for /dev/sdf1 :
UUID=F67C55827C553E95	/home/alex/Tainies	ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	00
#Entry for /dev/sdg1 :
UUID=E68EF2388EF200BB	/home/alex/White	ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	00
alex@HTPC:~
$ 
my inxi -Dxx --usb

Code: Select all

alex@HTPC:~
$ inxi -Dxx --usb
Drives:    HDD Total Size: 15.70 TiB used: 10.49 TiB (66.8%) 
           ID-1: /dev/sda vendor: Crucial model: CT250MX500SSD1 size: 232.89 GiB speed: 3.0 Gb/s 
           serial: 1815E1363D7C 
           ID-2: /dev/sdb vendor: Western Digital model: WD30EFRX-68AX9N0 size: 2.73 TiB speed: 3.0 Gb/s 
           serial: WD-WCC1T1239769 
           ID-3: /dev/sdc vendor: Samsung model: HD204UI size: 1.82 TiB speed: 3.0 Gb/s serial: S2K4JD2B500530 
           ID-4: /dev/sdd vendor: Samsung model: HD204UI size: 1.82 TiB speed: 3.0 Gb/s serial: S2H7J1CB206886 
           ID-5: /dev/sde type: USB vendor: Intenso model: External USB 3.0 size: 4.55 TiB 
           serial: 201701201003B 
           ID-6: /dev/sdf type: USB vendor: Samsung model: HD103SJ size: 931.51 GiB serial: S246J90Z165604 
           ID-7: /dev/sdg type: USB vendor: Hitachi model: HGST Touro Desk 3.0 size: 3.64 TiB 
           serial: 31001312180002300112 
           ID-8: /dev/sdh type: USB vendor: Kingston model: DataTraveler 3.0 size: 7.20 GiB 
           serial: 60A44C413985F12119750047 
USB:       Hub: 1:1 usb: 2.0 type: Full speed (or root) hub chip ID: 1d6b:0002 
           Hub: 2:1 usb: 3.0 type: Full speed (or root) hub chip ID: 1d6b:0003 
           Device-1: Samsung bus ID: 2:2 usb: 3.0 type: Mass Storage chip ID: 04e8:6113 
           Device-2: JMicron / JMicron USA bus ID: 2:3 usb: 3.0 type: Mass Storage chip ID: 152d:578f 
           Hub: 3:1 usb: 2.0 type: Full speed (or root) hub chip ID: 1d6b:0002 
           Hub: 3:3 usb: 2.0 type: Alcor Micro USB Hub chip ID: 058f:6254 
           Device-3: Cambridge Silicon Radio Bluetooth Dongle (HCI mode) bus ID: 3:5 usb: 2.0 type: Bluetooth 
           chip ID: 0a12:0001 
           Device-4: Seiko Epson bus ID: 3:6 usb: 2.0 type: Vendor Specific Protocol chip ID: 04b8:08d2 
           Device-5: Kingston DataTraveler G4 bus ID: 3:7 usb: 2.1 type: Mass Storage chip ID: 0951:1666 
           Hub: 4:1 usb: 2.0 type: Full speed (or root) hub chip ID: 1d6b:0002 
           Device-6: SimpleTech bus ID: 4:3 usb: 2.1 type: Mass Storage chip ID: 4971:1015 
           Hub: 5:1 usb: 1.1 type: Full speed (or root) hub chip ID: 1d6b:0001 
           Device-7: ROCCAT bus ID: 5:2 usb: 2.0 type: Keyboard chip ID: 1e7d:319c 
           Hub: 6:1 usb: 1.1 type: Full speed (or root) hub chip ID: 1d6b:0001 
           Device-8: Pixart Imaging Optical Mouse bus ID: 6:2 usb: 1.1 type: Mouse chip ID: 093a:2521 

User avatar
dolphin_oracle
Forum Veteran
Forum Veteran
Posts: 12142
Joined: Sun Dec 16, 2007 1:17 pm

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#16

Post by dolphin_oracle » Wed Jul 18, 2018 10:26 am

doing some filesystem research online I've found that in fact, fat32 formatted devices are measurably slower than the same device with a more modern format. could be part of the equation here.

scenario: file transfer appears fast until some cache or buffer on the device is full, then slows down to the device write speed. that's a guess.
http://www.youtube.com/runwiththedolphin
lenovo ThinkPad T530 - MX-18.3
lenovo s21e - antiX-19 beta 2 (live-USB)
FYI: mx "test" repo is not the same thing as debian testing repo.

User avatar
fehlix
Forum Veteran
Forum Veteran
Posts: 4580
Joined: Wed Apr 11, 2018 5:09 pm

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#17

Post by fehlix » Wed Jul 18, 2018 10:43 am

nutrialex wrote:
Wed Jul 18, 2018 10:18 am

Code: Select all

alex@HTPC:~
...
UUID=E68EF2388EF200BB	/home/alex/White	ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	00
...
Comment: yout fstab is a bit invalid: those line ending with '00' have to be ' 0 0'
So you might confuse the kernel a bit with this.

Code: Select all

alex@HTPC:~
...
           ID-8: /dev/sdh type: USB vendor: Kingston model: DataTraveler 3.0 size: 7.20 GiB 
           serial: 60A44C413985F12119750047 
..
          Device-5: Kingston DataTraveler G4 bus ID: 3:7 usb: 2.1 type: Mass Storage chip ID: 0951:1666 
Your Kingston-USB stick is plugged in or at least identified as USB 2.1 device,
which tells me you would not get much speed as USB2 is fairly slow.
So after your cache of 1GB is filled up with 80Mb/s it goes down to USB2 turtle speed of 3-5 Mb/s
Make sue you plugged in the kingston into USB3-port. After that do your tests again and show us fstab and inxi again.
Thanks
:turtle:
Gigabyte Z77M-D3H, Intel Xeon E3-1240 V2 (Quad core), 32GB RAM,
GeForce GTX 770, Samsung SSD 850 EVO 500GB, Seagate Barracuda 4TB

nutrialex
Forum Novice
Forum  Novice
Posts: 38
Joined: Wed Oct 21, 2015 3:40 am

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#18

Post by nutrialex » Wed Jul 18, 2018 11:22 am

I corrected fstab :

Code: Select all

alex@HTPC:~
$ cat  /etc/fstab
# /etc/fstab: static file system information
#
# Created by make-fstab on Wed Jul 18 03:36:44 EDT 2018

# <file system>                            <mount point>                 <type>     			<options>                       <dump/pass>

#-> /dev/sda1  label=MX17.1
UUID=8dbb220a-2c96-4c49-987f-beae063a41e7  /  		ext4   		defaults                   1 	1
#-> /dev/sda2  label=homeMX
UUID=e8ac3b1e-b614-4294-99d0-12bf57dbea57  /home 	ext4       	defaults                   1 	2

#Entry for /dev/sdb1 :
UUID=521C86F176C46EE1	/home/alex/3TB		ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8 	0 	0
#Entry for /dev/sdd1 :
UUID=EE98A92E98A8F66D	/home/alex/1		ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	0 	0
#Entry for /dev/sdc1 :
UUID=4AE86E39E86E2383	/home/alex/2		ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	0 	0
#Entry for /dev/sde1 :
UUID=656438C1331FFE55	/home/alex/5TB		ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	0 	0
#Entry for /dev/sdf1 :
UUID=F67C55827C553E95	/home/alex/Tainies	ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	0 	0
#Entry for /dev/sdg1 :
UUID=E68EF2388EF200BB	/home/alex/White	ntfs-3g	defaults,nodev,nosuid,locale=el_GR.UTF-8	0 	0

and I am very interested about what you said about cache because that is exactly what happens to me.
fehlix wrote:
Wed Jul 18, 2018 10:43 am

Your Kingston-USB stick is plugged in or at least identified as USB 2.1 device,
which tells me you would not get much speed as USB2 is fairly slow.
So after your cache of 1GB is filled up with 80Mb/s it goes down to USB2 turtle speed of 3-5 Mb/s
Make sue you plugged in the kingston into USB3-port. After that do your tests again and show us fstab and inxi again.
Sorry but I have not other usb3 left. There were 2 and there are to big drives there. So i unmount the 5TB and inserted the usb on the usb3 port. Speed is still very low as you described 1gb for 80MB/s and the rest with 2,5MB/s.

Here is the inxi:

Code: Select all

alex@HTPC:~
$ inxi -Dxx --usb
Drives:    HDD Total Size: 11.15 TiB used: 9.81 TiB (88.0%) 
           ID-1: /dev/sda vendor: Crucial model: CT250MX500SSD1 size: 232.89 GiB speed: 3.0 Gb/s 
           serial: 1815E1363D7C 
           ID-2: /dev/sdb vendor: Western Digital model: WD30EFRX-68AX9N0 size: 2.73 TiB 
           speed: 3.0 Gb/s serial: WD-WCC1T1239769 
           ID-3: /dev/sdc vendor: Samsung model: HD204UI size: 1.82 TiB speed: 3.0 Gb/s 
           serial: S2K4JD2B500530 
           ID-4: /dev/sdd vendor: Samsung model: HD204UI size: 1.82 TiB speed: 3.0 Gb/s 
           serial: S2H7J1CB206886 
           ID-5: /dev/sde type: USB vendor: Kingston model: DataTraveler 3.0 size: 7.20 GiB 
           serial: 60A44C413985F12119750047 
           ID-6: /dev/sdf type: USB vendor: Samsung model: HD103SJ size: 931.51 GiB 
           serial: S246J90Z165604 
           ID-7: /dev/sdg type: USB vendor: Hitachi model: HGST Touro Desk 3.0 size: 3.64 TiB 
           serial: 31001312180002300112 
USB:       Hub: 1:1 usb: 2.0 type: Full speed (or root) hub chip ID: 1d6b:0002 
           Hub: 2:1 usb: 3.0 type: Full speed (or root) hub chip ID: 1d6b:0003 
           Device-1: Samsung bus ID: 2:2 usb: 3.0 type: Mass Storage chip ID: 04e8:6113 
           Device-2: Kingston DataTraveler G4 bus ID: 2:4 usb: 3.1 type: Mass Storage 
           chip ID: 0951:1666 
           Hub: 3:1 usb: 2.0 type: Full speed (or root) hub chip ID: 1d6b:0002 
           Hub: 3:3 usb: 2.0 type: Alcor Micro USB Hub chip ID: 058f:6254 
           Device-3: Seiko Epson bus ID: 3:4 usb: 2.0 type: Vendor Specific Protocol 
           chip ID: 04b8:08d2 
           Device-4: Cambridge Silicon Radio Bluetooth Dongle (HCI mode) bus ID: 3:5 usb: 2.0 
           type: Bluetooth chip ID: 0a12:0001 
           Hub: 4:1 usb: 2.0 type: Full speed (or root) hub chip ID: 1d6b:0002 
           Device-5: SimpleTech bus ID: 4:3 usb: 2.1 type: Mass Storage chip ID: 4971:1015 
           Hub: 5:1 usb: 1.1 type: Full speed (or root) hub chip ID: 1d6b:0001 
           Device-6: ROCCAT bus ID: 5:2 usb: 2.0 type: Keyboard chip ID: 1e7d:319c 
           Hub: 6:1 usb: 1.1 type: Full speed (or root) hub chip ID: 1d6b:0001 
           Device-7: Pixart Imaging Optical Mouse bus ID: 6:2 usb: 1.1 type: Mouse 
           chip ID: 093a:2521 

clicktician
Forum Regular
Forum Regular
Posts: 227
Joined: Sat May 02, 2015 4:35 pm

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#19

Post by clicktician » Wed Jul 18, 2018 2:33 pm

nutrialex wrote:
Tue Jul 17, 2018 1:27 pm
Please someone post or find a solution. I do not want to go back to Mint. If I do it, it will be permanent.
Are you certain the overall write speed is slower? Do you know by how much?

The reason I mention this is, I just tried copying a 2GB video file to my Kingston Datatraveler USB Type C/ USB 3.1 drive (dual plugs) formatted as Fat32.
This drive is spec'd by the manufacturer at a MAX write speed of 15MB/sec. 15!

My junk.mp4 file is 1940.090023 MB in size, so it is physically impossible to copy this file to the Data Traveler stick in less than 129.34 seconds -- even ignoring any overhead.

The following command shows a run time of 35.92 seconds. An average of 54MB/sec, which we know cannot happen.

Code: Select all

time cp junk.mp4 /media/clicktician/dt/
It then takes ~98 seconds to eject the disk, flushing the cache to the drive. For a total of 133.9 seconds which is very close to the 15MB/sec max write speed of the stick.

When I turn off caching (note, you will have to install the nocache command: apt-get install nocache). It takes a whopping 329.875 seconds.

Code: Select all

time nocache cp junk.mp4 /media/clicktician/dt/
Obviously, caching greatly improves performance on many levels.

When I copy junk.mp4 with Thunar it shows it is copying at over 180MB/sec for the first 90% of the copy and then slows to 5Mb/sec. Total copy time is ~40 seconds. It takes another ~98 seconds to eject. This is consistent with the manufacturer's rated performance of the drive and my cp findings above.

It may be that the illusion that caching is creating for you, has caused you to believe something is wrong with your system.
Son, someday all this will belong to your ex wife.

User avatar
fehlix
Forum Veteran
Forum Veteran
Posts: 4580
Joined: Wed Apr 11, 2018 5:09 pm

Re: (MX17.1) Extremely low file copy speed to an external usb pen or disk (fat32 formatted)

#20

Post by fehlix » Wed Jul 18, 2018 3:24 pm

In addition to @clicktician's well described illusion scenario you need do some power consumption calculations as @stsoh already mentioned above. As you have lot's of usb-devices you need just sum up the potential power consumption with the power-source you provide to the devices.
When a drive starts with intensive writing , e.g after cache is filled up, the power consumption get to maximum and can reach for an usb3 device about 1000 mA , If not enough power is provided the device goes some gears down and you end up with slow write speed or even worse with write errors.
:snail:
Gigabyte Z77M-D3H, Intel Xeon E3-1240 V2 (Quad core), 32GB RAM,
GeForce GTX 770, Samsung SSD 850 EVO 500GB, Seagate Barracuda 4TB

Post Reply

Return to “Hardware /Configuration”