lz4 compression for mx

Message
Author
User avatar
Fornhamfred
Posts: 249
Joined: Sun May 02, 2010 12:49 pm

Re: lz4 compression for mx

#31 Post by Fornhamfred »

Eadwine Rose wrote:I would say it is only logical that it is a faster tool if it compresses less. More efficient compressing requires more time.

Personally.. compression should be proper compression. Not a half half solution that still leaves a big file. Might as well then copy it straight without compressing things... JMHO.
I have a working system of PcLinuxOS XFCE and just wondered how the figures stack up.
Using MyLiveGTK I got the following results on my system which the file size is 8.6GB
LZ4 Compressed to 4.17GB and took 8mins 6secs
ZX Compressed to 2.64GB and took 13 mins 33 secs
GZIP Compressed to 3.11GB and took 7mins 42 secs
The default compression ( not sure what it uses) Compressed to 2.40GB and took 29 mins 30 secs.

User avatar
Stevo
Developer
Posts: 12774
Joined: Fri Dec 15, 2006 8:07 pm

Re: lz4 compression for mx

#32 Post by Stevo »

Eadwine Rose wrote:Just wondering.. bit of a morbid bit of thinking here, but it has happened to me IRL the other way around, so... it is not unrealistic.


Suppose you compress the files.. pictures of yourself, your family, whatever... things you want to keep. Burn them to a dvd, usb, whatever rocks your boat. And then you die. Your house gets cleared out, hardware sold off, stuff put in boxes, storage.

And then a family member gets their hands on that disk, drive, whatever it was.. on there is written "photos 2018", and of course they want to have those!!

Can someone who uses windows still EASILY (remember we are talking basic users here) get to these files?

Compression is nice for software purposes, but there is the reason that I choose zip over other formats for some things.
Modern image formats like jpg or png are already compressed, as well as modern video formats, so it doesn't make sense to try and compress them again.

It could be a nice option in the snapshot tool to offer lz4, with the warning that the resulting image will be significantly larger. It's up to the user as to what's more important: size or time to compress.

Ubuntu may use yet another compression format now, zstd, that's supposed to make the install about 10% faster...and the ISOs about 6% larger: https://www.omgubuntu.co.uk/2018/03/fas ... ompression I can see that being a net loss...it adds more time to download and write the ISO to a medium than the few seconds you save when installing from a USB.

User avatar
Richard
Posts: 1577
Joined: Fri Dec 12, 2008 10:31 am

Re: lz4 compression for mx

#33 Post by Richard »

Lz4 compression worked very well in MX Snapshot. Two days ago made snapshot of recent dist-upgrade
with -comp xz which built a 2.933 GB ISO
This afternoon took snapshot
with -comp lz4 which built a 4.236 GB ISO in 3 min

The new ≥4GB ISO created a usable USB with Live-USB-Maker, but failed to boot on Easy2Boot USB recreated with ntfs for the oversized ISO.

Tried again with -comp -5 lz4 but Snapshot couldn't handle the option. Tried again with Snapshots default compression --gz. And it works as expected on Easy2Boot.
Last edited by Richard on Sun Mar 18, 2018 8:23 pm, edited 2 times in total.
Thinkpad T430 & Dell Latitude E7450, both with MX-21.3.1
kernal 5.10.0-26-amd64 x86_64; Xfce-4.18.0; 8 GB RAM
Intel Core i5-3380M, Graphics, Audio, Video; & SSDs.

skidoo
Posts: 753
Joined: Tue Sep 22, 2015 6:56 pm

Re: lz4 compression for mx

#34 Post by skidoo »

.
package in debian stable (Stretch) is v4.3-3 ...but contains an outdated manpage, written in 2012
https://sources.debian.org/src/squashfs ... hfs.1/#L25
Debian's buster/sid mksquashfs package also contains that same (2012, outdated) manpage
https://sources.debian.org/src/squashfs ... quashfs.1/
^--- significant b/c the older manpage lacks mention of mksquashfs support for lz4, and ability to preserve extended attributes

In the absence of an up-do-date manpage, the following page at author's github repo stand as the definitive documentation:
https://github.com/plougher/squashfs-to ... ASE-README

==============

an undocumented mksquashfs feature:
ref: https://github.com/plougher/squashfs-tools/issues/24 and action.c
We can pass one or multiple -action commandline options to mksquashfs,
to specify wildcard (or regex) patterns and instruct "don't waste effort attempting to re-compress files matching this pattern"
^--- significant regardless which compression algorithm is chosen (for instance, filesystem contains hundreds of .gz manpage files...)

==============

(mentioned for the sake of completeness)
another undocumented, or at least rarely mentioned, mksquashfs feature is ability to create/inject files "out of thin air":
https://github.com/plougher/squashfs-to ... le.example

==============

In addition to lz4, mksquashfs will {now} soon provide ability to choose zstd compression. (ref: wikipedia and here)
(FWIW, zstd -compressed images can be consumed by the 4.14 kernel and squashFUSE)
^--- the code now is in place, pending a new squashfs-tools version release

==============

anecdotal: Last year (I wasn't aware of lz4 availability at the time) I conducted experiments, varying the levels of compression and the various compressors. Applying lighter compression when creating an "antiX full" snapshot yielded 2sec (newer PC) to 4sec (older PC) shorter boot times.
Ubuntu may use yet another compression format now, zstd, that's supposed to make the install about 10% faster...and the ISOs about 6% larger: I can see that being a net loss...it adds more time to download and write the ISO to a medium than the few seconds you save when installing from a USB.
aw, I drafted this while offline & didn't notice that Stevo had already mentioned zstd.

User avatar
Richard
Posts: 1577
Joined: Fri Dec 12, 2008 10:31 am

Re: lz4 compression for mx

#35 Post by Richard »

Wow!
lz4 is several orders of magnitude faster. at least 3, on antiX-17 on Acer Aspire One, 386 netbook with 150 GB HDD. Took less than 10 minutes to complete. [Usually takes about 30 to 40 minutes]. Generated ISO is about 62% larger.
Seems to take as long to copy to a USB as it did to make the snapshot. :)
Thinkpad T430 & Dell Latitude E7450, both with MX-21.3.1
kernal 5.10.0-26-amd64 x86_64; Xfce-4.18.0; 8 GB RAM
Intel Core i5-3380M, Graphics, Audio, Video; & SSDs.

bled
Posts: 58
Joined: Mon Mar 07, 2016 1:06 pm

xz and gz overheat computer

#36 Post by bled »

hi richard

can you investigate the level of temperature
of computer during remaster process?
in xz and gz temp. level is very high -up to 70 degree c,
in lz4 is about 4o degree c.
xz and gz overheat computer...
regards

User avatar
Richard
Posts: 1577
Joined: Fri Dec 12, 2008 10:31 am

Re: lz4 compression for mx

#37 Post by Richard »

I only observed but confirmed that lz4 ran much cooler than xz when running on mx17x64 on my laptop. Max temp apprx 60°C compared to 95°C with xz. Resulting ISO size of apprx 4.37 GB worked fine building LiveUSB with MX-Live-USB-Maker.

On antiX-17, lz4 ran much cooler, quicker on 32bit netbook and resulting ISO built and executed as expected. Only observable temp on this installation was 58°C, considerably cooler than before.

They both ran so much quicker so probably generated less heat. I will make notes next time.
Last edited by Richard on Mon Mar 19, 2018 7:54 am, edited 4 times in total.
Thinkpad T430 & Dell Latitude E7450, both with MX-21.3.1
kernal 5.10.0-26-amd64 x86_64; Xfce-4.18.0; 8 GB RAM
Intel Core i5-3380M, Graphics, Audio, Video; & SSDs.

User avatar
richb
Administrator
Posts: 10322
Joined: Wed Jul 12, 2006 2:17 pm

Re: lz4 compression for mx

#38 Post by richb »

Not sure about temps as my AMD does not report absolute temperatures but uses a reference temperature. However gz utilizes 100 % CPU while lz4 averages about 40% during a snapshot creation. The resulting iso using lz4 boots without a problem.
Forum Rules
Guide - How to Ask for Help

richb Administrator
System: MX 23 KDE
AMD A8 7600 FM2+ CPU R7 Graphics, 16 GIG Mem. Three Samsung EVO SSD's 250 GB

User avatar
stsoh
Posts: 183
Joined: Sun Aug 20, 2017 10:11 am

Re: lz4 compression for mx

#39 Post by stsoh »

lz4 high compression (only one option in snapshot, default is fast compression).

Code: Select all

mksq_opt=-comp lz4 -Xhc
reduce compressed file size from 3.9 to 3.4gb, takes the same time to complete snapshot task.
MX-17.1_x64 Horizon, G41M-P33 Combo (MS-7592), Pentium E5400 (2706 MHz), 8Gb RAM (984 MT/s),
Intel 4 Series Integrated Graphics, Realtek PCIe Fast RTL8101/2/6E, PCI Gigabit RTL8169 Ethernets.
Accepted Linux when i found MX-Linux in 2016.

User avatar
Stevo
Developer
Posts: 12774
Joined: Fri Dec 15, 2006 8:07 pm

Re: lz4 compression for mx

#40 Post by Stevo »

The high compression lz4 sounds like a very attractive option for mx-snapshot. Is lz4 single-threaded only, and that's why it doesn't use 100% of the CPU on multicore/hyperthreaded systems like xz does while snapshotting? There's a XFCE cpugraph panel plugin that'll let you watch your cores, or some conky themes also allow this. If it supports parallel compression, would that speed up the snapshot process even more?

The Liquorix kernels we've been porting have used lz4 since 4.12. I'm not sure what this means in the Debian kernel changelog, but we have it in the 4.14 and 4.15 ports we've also done:

Code: Select all

  [ Vagrant Cascadian ]
  * Enable SQUASHFS_LZ4 in default config.

Post Reply

Return to “General”