mx-live-usb-maker inconsistency

[greywolf]

Not sure if everyone would call this a 'bug' or not? The application still works but I found an inconsistency in its operation.

I had occasion to use the app to produce a live stick from an iso from other than mx/antix; pclinuxos-xfce to be precise.

I understand it reverts to a front end for the 'dd' command in this scenario (having chosen that option with a tick). The process worked but the gui has an issue whereby it came back with the "Write Completed" message LONG before the write has actually finished. The message returned within 10-20 seconds and I knew that was way to quick for a write of some 1.2Gb. Luckily the stick I was using has an activity light on it which was still flashing, so I just let it go until it stopped and it was more like the few minutes such a write would take from the command line. I even left it a bit longer and it gave a final flicker of both the usb stick light and the hard drive light. I am assuming that was the 'real' process finish.

I ejected the stick from cli and tested it. It works ok so the process worked. However, if the stick had not had an activity light, and many do not, I could have easily seen the completed message and removed the stick way before the write had finished. Obviously creating a stick that would be broken. If this is not unique to me, perhaps someone could take a look at the app's workings. I assume it is not seeing some condition feedback correctly.

In the meantime I will leave these convoluted gui thingys alone and revert to my trusty old cli for dd commands!

greywolf.

[chrispop99]

Just to clarify, within the GUI you made sure the tick-box for 'Copy image to USB (dd)' was ticked?

I have used that method for non-MX ISO's with complete success.

Chris

[greywolf]

Just to clarify, within the GUI you made sure the tick-box for 'Copy image to USB (dd)' was ticked?

Yes Chris, that's what I meant by:

I understand it reverts to a front end for the 'dd' command in this scenario (having chosen that option with a tick).

I have used that method for non-MX ISO's with complete success.

Well, I guess that's why I asked if it is in fact a "bug"? Technically, the process worked in the sense that I got a workable usb stick. BUT, I may easily have not had success if I had removed the stick too early, believing the message that the write had successfully completed when it clearly had not. I might do some extra testing with other sticks and with other isos?

thanks,
greywolf

[chrispop99]

Further testing is always welcome if you are able to do so - thanks.

Chris

[dolphin_oracle]

this is the command used by mx-lum

dd bs=1M if=source of=/dev/usb_device


should there be another parameter?

[Adrian]

I'm not sure what's going on, if the dd command reports it ended and the USB is still written maybe it means it just finished writing to the buffer and the buffer is still written on the USB drive?
While the progress bar can possibly be wrong the box that says "Process done" should show up only when the dd command finishes. Did you see that pop up message or only looked at the progress bar?

[BitJam]

The process worked but the gui has an issue whereby it came back with the "Write Completed" message LONG before the write has actually finished. The message returned within 10-20 seconds and I knew that was way to quick for a write of some 1.2Gb.

This is basically a long standing kernel bug. We are fighting back against it. The problem is that the Kernel does WAY too much buffering. So if a process is IO-bound and not CPU-bound then the process will finish quickly because it wrote the output to a huge ram buffer and it has nothing left to do so it exits. Then our program runs the "sync" command which waits until the writes actually make it to the disk. This is the delay you are seeing.

I have a good fix for this that I've implemented in the cli dd-live-usb and the cli live-usb-maker (and hence in the standard gui version of live-usb-maker). The only one left out is the gui version of dd-live-usb which is separate code. The fix is I set vm.dirty_bytes to a much smaller value. This means we sync as we go along so the progress meter measuring file size become much more accurate and there is no big wait for the final sync when it is done. An added benefit, at least for usb sticks, is this trick makes the transfer 5% to 10% faster. You can test this by timing your manual way with "time dd ..." and comparing it with the time reported by dd-live-usb. The first time result should be tossed because it may take longer if it is caching the file you are reading. You can also get around this by running "echo 3 | sudo tee /proc/sys/vm/drop_caches" before each "dd". this will ensure the file is not cached, which is actually a better simulation of the real world.

I just added this trick to live-remaster. The compression had always been cpu-bound so this was never an issue (on my hardware) but when we added the super-fast lz4 compression, the process became IO-bound so there was suddenly a big wait after the progress bar said it was done and we waited about 30 seconds as sync waited for the file to actually get written. The vm.dirty_bytes trick fixed this.

@Dolphin. This is really easy, especially if you are already using cli-shell-utils. But if you're not, it is just a simple copy and paste like what I just did in live-remaster. Once the boiler plate code is sourced or copied then:

1. Add "set_dirty_bytes" before the dd
2. Add "restore_dirty_bytes_and_ratio" after the sync
3. Make sure "restore_dirty_bytes_and_ratio" is called by your exit routine

[dolphin_oracle]

I forgot about dirty.bytes.

I used this in lum-gui

Code: Select all

 #adjust vm.dirty.bytes for progress meter and speed bump
    ORIG_DIRTY_RATIO=$(sysctl -n vm.dirty_ratio)
    ORIG_DIRTY_BYTES=$(sysctl -n vm.dirty_bytes)
    usb_dirty_bytes_gui=20000000
    sysctl vm.dirty_bytes=$usb_dirty_bytes_gui >> $LOG_FILE

then ran the dd operation, and after the end

Code: Select all

#put vm.dirty.bytes back the way we found theme
    sysctl vm.dirty_bytes=$ORIG_DIRTY_BYTES >> $LOG_FILE
    sysctl vm.dirty_ratio=$ORIG_DIRTY_RATIO >> $LOG_FILE
    unset ORIG_DIRTY_BYTES ORIG_DIRTY_RATIO

and then this in the "clean up" code

Code: Select all

#reset dirty_bytes if we changed it and we have an unexpected exit
    [ "$ORIG_DIRTY_BYTES" ] && sysctl vm.dirty_bytes=$ORIG_DIRTY_BYTES >> $LOG_FILE
    [ "$ORIG_DIRTY_RATIO" ] && sysctl vm.dirty_ratio=$ORIG_DIRTY_RATIO >> $LOG_FILE
    

[BitJam]

I forgot about dirty.bytes.

I used this in lum-gui [...]

If you were already using that code then it should be working correctly. Did you just add this code?

Is it possible you say you are done before the "sync"? That still wouldn't explain why dirty.bytes isn't working for greywolf. You can run "sysctl -n vm.dirty_bytes" while the dd is taking place to see if it has the right value.

It was working here correctly very recently on antiX-17.1 when I was adding lz4 to live-remaster. The time for the sync fell from 30 secs down to 0.5 secs.

Is it possible greywolf is using an earlier version that doesn't set dirty_bytes?

[Adrian]

I don't think we use that in mx-lum, I think D_O meant he used that in lum-gui which was the old GUI. We'll have to implement that or simply add && sync after the dd command to make sure everything is synced before we pop-up the message "Process done"