Commit 05a1b863 authored by longpanda's avatar longpanda
Browse files

initial commit

parent 2090c6fa
ACKNOWLEDGEMENTS
Thanks to everyone who have downloaded Squashfs. I appreciate people
using it, and any feedback you have.
The following have provided useful feedback, which has guided
some of the extra features in squashfs. This is a randomly ordered
(roughly in chronological order) list, which is updated when
I remember...
Acknowledgements for Squashfs 4.3
---------------------------------
Thanks to Bruno Wolff III and Andy Lutomirski for useful feedback
during the long development process of Squashfs 4.3.
Acknowledgements for Squashfs 4.2
---------------------------------
Thanks to Lasse Collin (http://tukaani.org/xz/) for mainlining XZ
decompression support.
Acknowledgements for Squashfs 4.1
---------------------------------
Thanks to Chan Jeong <chan.jeong@lge.com> and LG for the patches to support LZO
compression.
Acknowledgements for Squashfs 4.0
---------------------------------
Thanks to Tim Bird and CELF (Consumer Electronics Linux Forum) for helping
fund mainstreaming of Squashfs into the 2.6.29 kernel and the
changes to the Squashfs tools to support the new 4.0 file system layout.
Acknowledgements for Squashfs-3.3
------------------------------------
Peter Korsgaard and others sent patches updating Squashfs to changes in the
VFS interface for 2.6.22/2.6.23/2.6.24-rc1. Peter also sent some small patches
for the Squashfs kernel code.
Vito Di Leo sent a patch extending Mksquashfs to support regex filters.
While his patched worked, it unfortunately made it easy to make Mksquashfs
perform unpredictably with poorly choosen regex expressions. It, however,
encouraged myself to add support for wildcard pattern matching and regex
filters in a different way.
Acknowledgements for Squashfs-3.2-r2
------------------------------------
Junjiro Okajima discovered a couple of SMP issues, thanks.
Junjiro Okajima and Tomas Matejicek have produced some good LZMA patches
for Squashfs.
Acknowledgements for Squashfs-3.2
---------------------------------
Peter Korsgaard sent a patch updating Squashfs to changes in the VFS interface
in Linux 2.6.20.
Acknowledgements for Squashfs-3.1
---------------------------------
Kenneth Duda and Ed Swierk of Arastra Inc. identified numerous bugs with
Squashfs, and provided patches which were the basis for some of the
fixes. In particular they identified the fragment rounding bug, the
NFS bug, the initrd bug, and helped identify the 4K stack overflow bug.
Scott James Remnant (Ubuntu) also identified the fragment rounding bug,
and he also provided a patch.
Ming Zhang identified the Lseek bug in Mksquashfs. His tests on the
performance of Mksquashfs on SMP systems encouraged the rewrite of
Mksquashfs.
Peter Korsgaard, Daniel Olivera and Zilvinas Valinskas noticed
Squashfs 3.0 didn't compile on Linux-2.6.18-rc[1-4] due to changes
in the Linux VFS interfaces, and provided patches.
Tomas Matejicek (SLAX) suggested the -force option on Unsquashfs, and noticed
Unsquashfs didn't return the correct exit status.
Yann Le Doare reported a kernel oops and provided a Qemu image that led
to the identification of the simultaneously accessing multiply mounted Squashfs
filesystems bug.
Older acknowledgements
----------------------
Mark Robson - pointed out early on that initrds didn't work
Adam Warner - pointed out that greater than 2GB filesystems didn't work.
John Sutton - raised the problem when archiving the entire filesystem
(/) there was no way to prevent /proc being archived. This prompted
exclude files.
Martin Mueller (LinuxTV) - noticed that the filesystem length in the
superblock doesn't match the output filesystem length. This is due to
padding to a 4K boundary. This prompted the addition of the -nopad option.
He also reported a problem where 32K block filesystems hung when used as
initrds.
Arkadiusz Patyk (Polish Linux Distribution - PLD) reported a problem where 32K
block filesystems hung when used as a root filesystem mounted as a loopback
device.
David Fox (Lindows) noticed that the exit codes returned by Mksquashfs were
wrong. He also noticed that a lot of time was spent in the duplicate scan
routine.
Cameron Rich complained that Squashfs did not support FIFOs or sockets.
Steve Chadsey and Thomas Weissmuller noticed that files larger than the
available memory could not be compressed by Mksquashfs.
"Ptwahyu" and "Hoan" (I have no full names and I don't like giving people's
email addresses), noticed that Mksquashfs 1.3 SEGV'd occasionally. Even though
I had already noticed this bug, it is useful to be informed by other people.
Don Elwell, Murray Jensen and Cameron Rich, have all sent in patches. Thanks,
I have not had time to do anything about them yet...
Drew Scott Daniels has been a good advocate for Squashfs.
Erik Andersen has made some nice suggestions, unfortunately, I have
not had time to implement anything.
Artemiy I. Pavlov has written a useful LDP mini-howto for Squashfs
(http://linuxdoc.artemio.net/squashfs).
Yves Combe reported the Apple G5 bug, when using Squashfs for
his PPC Knoppix-mib livecd project.
Jaco Greeff (mklivecd project, and maintainer of the Mandrake
squashfs-tools package) suggested the new mksquashfs -ef option, and the
standalone build for mksquashfs.
Mike Schaudies made a donation.
Arkadiusz Patyk from the Polish Linux Distribution reported that Squashfs
didn't work on amd64 machines. He gave me an account on a PLD amd64 machine
which allowed myself to track down these bugs.
Miles Roper, Peter Kjellerstedt and Willy Tarreau reported that release 2.1 did
not compile with gcc < 3.x.
Marcel J.E. Mol reported lack of kernel memory issues when using Squashfs
on small memory embedded systems. This prompted the addition of the embedded
system kernel configuration options.
Era Scarecrow noticed that Mksquashfs had not been updated to reflect that
smaller than 4K blocks are no longer supported.
Kenichi Shima reported the Kconfig file had not been updated to 2.2.
Aaron Ten Clay made a donation!
Tomas Matejicek (SLAX) made a donation!
SQUASHFS CHANGE LOG
4.4 29 AUG 2019 Reproducible builds, new compressors,
CVE fixes, security hardening and new options
for Mksquashfs/Unsquashfs.
1. Overall improvements:
1.1 Mksquashfs now generates reproducible images by default.
1.2 Mkfs time and file timestamps can also be specified.
1.3 Support for the Zstandard (ZSTD) compression algorithm.
1.4 CVE-2015-4645 and CVE-2015-4646 have been fixed.
2. Mksquashfs improvements and major bug fixes:
2.1 Pseudo files now support symbolic links.
2.2 New -mkfs-time option.
2.3 New -all-time option.
2.4 New -root-mode option.
2.5 New -quiet option.
2.6 New -noId option.
2.7 New -offset option.
2.8 Update lz4 wrapper to use new functions introduced
in 1.7.0.
2.9 Bug fix, don't allow "/" pseudo filenames.
2.10 Bug fix, allow quoting of pseudo files, to
better handle filenames with spaces.
2.11 Fix compilation with glibc 2.25+.
3. Unsquashfs improvements and major bug fixes:
3.1 CVE-2015-4645 and CVE-2015-4646 have been fixed.
3.2 Unsquashfs has been further hardened against corrupted
filestems.
3.3 Unsquashfs is now more strict about error handling.
3.4 New -ignore-errors option.
3.5 New -strict-errors option.
3.6 New -lln[umeric] option.
3.7 New -lc option.
3.8 New -llc option.
3.9 New -mkfs-time option.
3.10 New -UTC option.
3.11 New -offset option.
3.12 New -quiet option.
3.13 Update lz4 wrapper to use new functions introduced
in 1.7.0.
3.14 Bug fix, fatal and non-fatal errors now set the exit
code to 1.
3.15 Bug fix, fix time setting for symlinks.
3.16 Bug fix, try to set sticky-bit when running as a
user process.
3.17 Fix compilation with glibc 2.25+.
4.3 12 MAY 2014 New compressor options, new Mksquashfs/Unsquashfs
functionality, duplicate checking optimisations,
stability improvements (option/file parsing,
buffer/memory overflow checks, filesystem hardening
on corrupted filesystems), CVE fixes.
Too many changes to do the traditional custom changelog. But, this
is now unnecessary, so instead list most significant 15% of commits
from git changelog in chronological order.
- unsquashfs: add checks for corrupted data in opendir functions
- unsquashfs: completely empty filesystems incorrectly generate an error
- unsquashfs: fix open file limit
- mksquashfs: Use linked list to store directory entries rather
- mksquashfs: Remove qsort and add a bottom up linked list merge sort
- mksquashfs: optimise lookup_inode2() for dirs
- pseudo: fix handling of modify pseudo files
- pseudo: fix handling of directory pseudo files
- xattr: Fix ERROR() so that it is synchronised with the progress bar
- mksquashfs/sort: Fix INFO() so that it is synced with the progress bar
- mksquashfs: Add -progress to force progress bar when using -info
- error.h: consolidate the various error macros into one header file
- mksquashfs: fix stack overflow in write_fragment_table()
- mksquashfs: move list allocation from off the stack
- unsquashfs: fix oversight in directory permission setting
- mksquashfs: dynamically allocate recovery_file
- mksquashfs: dynamically allocate buffer in subpathname()
- mksquashfs: dynamically allocate buffer in pathname()
- unsquashfs: fix CVE-2012-4024
- unsquashfs: fix CVE-2012-4025
- mksquashfs: fix potential stack overflow in get_component()
- mksquashfs: add parse_number() helper for numeric command line options
- mksquasfs: check return value of fstat() in reader_read_file()
- mksquashfs: dynamically allocate filename in old_add_exclude()
- unsquashfs: dynamically allocate pathname in dir_scan()
- unsquashfs: dynamically allocate pathname in pre_scan()
- sort: dynamically allocate filename in add_sort_list()
- mksquashfs: fix dir_scan() exit if lstat of source directory fails
- pseudo: fix memory leak in read_pseudo_def() if exec_file() fails
- pseudo: dynamically allocate path in dump_pseudo()
- mksquashfs: dynamically allocate path in display_path2()
- mksquashfs: dynamically allocate b_buffer in getbase()
- pseudo: fix potential stack overflow in get_component()
- pseudo: avoid buffer overflow in read_pseudo_def() using sscanf()
- pseudo: dynamically allocate filename in exec_file()
- pseudo: avoid buffer overflow in read_sort_file() using fscanf()
- sort: tighten up sort file parsing
- unsquashfs: fix name under-allocation in process_extract_files()
- unsquashfs: avoid buffer overflow in print_filename() using sprintf()
- Fix some limits in the file parsing routines
- pseudo: Rewrite pseudo file processing
- read_fs: fix small memory leaks in read_filesystem()
- mksquashfs: fix fclose leak in reader_read_file() on I/O error
- mksquashfs: fix frag struct leak in write_file_{process|blocks|frag}
- unsquashfs_xattr: fix memory leak in write_xattr()
- read_xattrs: fix xattr free in get_xattr() in error path
- unsquashfs: add -user-xattrs option to only extract user.xxx xattrs
- unsquashfs: add code to only print "not superuser" error message once
- unsquashfs: check for integer overflow in user input
- mksquashfs: check for integer overflow in user input
- mksquashfs: fix "new" variable leak in dir_scan1()
- read_fs: prevent buffer {over|under}flow in read_block() with
corrupted filesystems
- read_fs: check metadata blocks are expected size in scan_inode_table()
- read_fs: check the root inode block is found in scan_inode_table()
- read_fs: Further harden scan_inode_table() against corrupted
filesystems
- unsquashfs: prevent buffer {over|under}flow in read_block() with
corrupted filesystems
- read_xattrs: harden xattr data reading against corrupted filesystems
- unsquash-[23]: harden frag table reading against corrupted filesystems
- unsquash-4.c: harden uid/gid & frag table reading against corruption
- unsquashfs: harden inode/directory table reading against corruption
- mksquashfs: improve out of space in output filesystem handling
- mksquashfs: flag lseek error in writer as probable out of space
- mksquashfs: flag lseek error in write_destination as probable out of
space
- mksquashfs: print file being squashed when ^\ (SIGQUIT) typed
- mksquashfs: make EXIT_MKSQUASHFS() etc restore via new restore thread
- mksquashfs: fix recursive restore failure check
- info: dump queue and cache status if ^\ hit twice within one second
- mksquashfs: fix rare race condition in "locked fragment" queueing
- lz4: add experimental support for lz4 compression
- lz4: add support for lz4 "high compression"
- lzo_wrapper: new implementation with compression options
- gzip_wrapper: add compression options
- mksquashfs: redo -comp <compressor> parsing
- mksquashfs: display compressor options when -X option isn't recognised
- mksquashfs: add -Xhelp option
- mksquashfs/unsquashfs: fix mtime signedness
- Mksquashfs: optimise duplicate checking when appending
- Mksquashfs: introduce additional per CPU fragment process threads
- Mksquashfs: significantly optimise fragment duplicate checking
- read_fs: scan_inode_table(), fix memory leak on filesystem corruption
- pseudo: add_pseudo(), fix use of freed variable
- mksquashfs/unsquashfs: exclude/extract/pseudo files, fix handling of
leaf name
- mksquashfs: rewrite default queue size so it's based on physical mem
- mksquashfs: add a new -mem <mbytes> option
- mksquashfs: fix limit on the number of dynamic pseudo files
- mksquashfs: make -mem take a normal byte value, optionally with a
K, M or G
4.2 28 FEB 2011 XZ compression, and compression options support
1. Filesystem improvements:
1.1 Added XZ compression
1.2 Added compression options support
2. Miscellaneous improvements/bug fixes
1.1 Add missing NO_XATTR filesystem flag to indicate no-xattrs
option was specified and no xattrs should be stored when
appending.
1.2 Add suppport in Unquashfs -stat option for displaying
NO_XATTR flag.
1.3 Remove checkdata entry from Unsquashfs -stat option if a 4.0
filesystem - checkdata is no longer supported.
1.4 Fix appending bug when appending to an empty filesystem - this
would be incorrectly treated as an error.
1.5 Use glibc sys/xattr.h include rather than using attr/xattr.h
which isn't present by default on some distributions.
1.6 Unsquashfs, fix block calculation error with regular files when
file size is between 2^32-block_size+1 and 2^32-1.
1.7 Unsquashfs, fix sparse file writing when holes are larger than
2^31-1.
1.8 Add external CFLAGS and LDFLAGS support to Makefile, and allow
build options to be specified on command line. Also don't
over-write passed in CFLAGS definition.
4.1 19 SEPT 2010 Major filesystem and tools improvements
1. Filesystem improvements:
1.1 Extended attribute support
1.2 New compression framework
1.3 Support for LZO compression
1.4 Support for LZMA compression (not yet in mainline)
2. Mksquashfs improvements:
1.1 Enhanced pseudo file support
1.2 New options for choosing compression algorithm used
1.3 New options for controlling extended attributes
1.4 Fix misalignment issues with memcpy etc. seen on ARM
1.5 Fix floating point error in progress_bar when max == 0
1.6 Removed use of get_nproc() call unavailable in ulibc
1.7 Reorganised help text
3. Unsquashfs improvements:
1.1 New options for controlling extended attributes
1.2 Fix misalignment issues with memcpy etc. seen on ARM
1.3 Fix floating point error in progress_bar when max == 0
1.4 Removed use of get_nproc() call unavailable in ulibc
4.0 5 APR 2009 Major filesystems improvements
1. Kernel code improvements:
1.1 Fixed little endian layout adopted. All swapping macros
removed, and in-line swapping added for big-endian
architectures.
1.2 Kernel code substantially improved and restructured.
1.3 Kernel code split into separate files along functional lines.
1.4 Vmalloc usage removed, and code changed to use separately
allocated 4K buffers
2. Unsquashfs improvements:
2.1 Support for 4.0 filesystems added.
2.2 Swapping macros rewritten.
2.3 Unsquashfs code restructured and split into separate files.
3. Mksquashfs improvements:
3.1 Swapping macros rewritten. Fixed little-endian layout allows
code to be optimised and only added at compile time for
big endian systems.
3.2 Support for pseudo files added.
3.4 26 AUG 2008 Performance improvements to Unsquashfs, Mksquashfs
and the kernel code. Plus many small bug fixes.
1. Kernel code improvements:
1.1 Internal Squashfs kernel metadata and fragment cache
implementations have been merged and optimised. Spinlocks are
now used, locks are held for smaller periods and wakeups have
been minimised. Small race condition fixed where if two or
more processes tried to read the same cache block
simultaneously they would both read and decompress it. 10-20%+
speed improvement has been seen on tests.
1.2 NFS export code rewritten following VFS changes in
linux-2.6.24.
1.3 New patches for linux-2.6.25, linux-2.6.26, and linux-2.6.27.
Fixed patch for linux-2.6.24.
1.4 Fixed small buffer_head leak in squashfs_read_data when
handling badly corrupted filesystems.
1.5 Fixed bug in get_dir_index_using_offset.
2. Unsquashfs improvements:
2.1 Unsquashfs has been parallelised. Filesystem reading, writing
and decompression is now multi-threaded. Up to 40% speed
improvement seen on tests.
2.2 Unsquashfs now has a progress bar. Use -no-progress to
disable it.
2.3 Fixed small bug where unistd.h wasn't being included on
some distributions, leading to lseek being used rather than
lseek64 - which meant on these distributions Unsquashfs
couldn't unsquash filesystems larger than 4GB.
3. Mksquashfs improvements:
3.1 Removed some small remaining parallelisation bottlenecks.
Depending on source filesystem, up to 10%+ speed improvement.
3.2 Progress bar improved, and moved to separate thread.
3.3 Sparse file handling bug in Mksquashfs 3.3 fixed.
3.4 Two rare appending restore bugs fixed (when ^C hit twice).
3.3 1 NOV 2007 Increase in block size, sparse file support,
Mksquashfs and Unsquashfs extended to use
pattern matching in exclude/extract files, plus
many more improvements and bug fixes.
1. Filesystem improvements:
1.1. Maximum block size has been increased to 1Mbyte, and the
default block size has been increased to 128 Kbytes.
This improves compression.
1.2. Sparse files are now supported. Sparse files are files
which have large areas of unallocated data commonly called
holes. These files are now detected by Squashfs and stored
more efficiently. This improves compression and read
performance for sparse files.
2. Mksquashfs improvements:
2.1. Exclude files have been extended to use wildcard pattern
matching and regular expressions. Support has also been
added for non-anchored excludes, which means it is
now possible to specify excludes which match anywhere
in the filesystem (i.e. leaf files), rather than always
having to specify exclude files starting from the root
directory (anchored excludes).
2.2. Recovery files are now created when appending to existing
Squashfs filesystems. This allows the original filesystem
to be recovered if Mksquashfs aborts unexpectedly
(i.e. power failure).
3. Unsquashfs improvements:
3.1. Multiple extract files can now be specified on the
command line, and the files/directories to be extracted can
now also be given in a file.
3.2. Extract files have been extended to use wildcard pattern
matching and regular expressions.
3.3. Filename printing has been enhanced and Unquashfs can
now display filenames with file attributes
('ls -l' style output).
3.4. A -stat option has been added which displays the filesystem
superblock information.
3.5. Unsquashfs now supports 1.x filesystems.
4. Miscellaneous improvements/bug fixes:
4.1. Squashfs kernel code improved to use SetPageError in
squashfs_readpage() if I/O error occurs.
4.2. Fixed Squashfs kernel code bug preventing file
seeking beyond 2GB.
4.3. Mksquashfs now detects file size changes between
first phase directory scan and second phase filesystem create.
It also deals better with file I/O errors.
3.2-r2 15 JAN 2007 Kernel patch update and progress bar bug fix
1. Kernel patches 2.6.19/2.6.20 have been updated to use
const structures and mutexes rather than older semaphores.
2. Minor SMP bug fixes.
3. Progress bar broken on x86-64. Fixed.
3.2 2 JAN 2007 NFS support, improvements to the Squashfs-tools, major
bug fixes, lots of small improvements/bug fixes, and new
kernel patches.
Improvements:
1. Squashfs filesystems can now be exported via NFS.
2. Unsquashfs now supports 2.x filesystems.
3. Mksquashfs now displays a progress bar.
4. Squashfs kernel code has been hardened against accidently or
maliciously corrupted Squashfs filesystems.
Bug fixes:
5. Race condition occurring on S390 in readpage() fixed.
6. Odd behaviour of MIPS memcpy in read_data() routine worked-around.
7. Missing cache_flush in Squashfs symlink_readpage() added.
3.1-r2 30 AUG 2006 Mksquashfs -sort bug fix
A code optimisation after testing unfortunately
broke sorting in Mksquashfs. This has been fixed.
3.1 19 AUG 2006 This release has some major improvements to
the squashfs-tools, a couple of major bug
fixes, lots of small improvements/bug fixes,
and new kernel patches.
1. Mksquashfs has been rewritten to be multi-threaded. It
has the following improvements
1.1. Parallel compression. By default as many compression and
fragment compression threads are created as there are available
processors. This significantly speeds up performance on SMP
systems.
1.2. File input and filesystem output is peformed in parallel on
separate threads to maximise I/O performance. Even on single
processor systems this speeds up performance by at least 10%.
1.3. Appending has been significantly improved, and files within the
filesystem being appended to are no longer scanned and
checksummed. This significantly improves append time for large
filesystems.
1.4. File duplicate checking has been optimised, and split into two
separate phases. Only files which are considered possible
duplicates after the first phase are checksummed and cached in
memory.
1.5 The use of swap memory was found to significantly impact
performance. The amount of memory used to cache files is now a
command line option, by default this is 512 Mbytes.
2. Unsquashfs has the following improvements
2.1 Unsquashfs now allows you to specify the filename or the
directory within the Squashfs filesystem that is to be
extracted, rather than always extracting the entire filesystem.
2.2 A new -force option has been added which forces Unsquashfs to
output to the destination directory even if files and directories
already exist in the destination directory. This allows you to
update an already existing directory tree, or to Unsquashfs to
a partially filled directory tree. Without the -force option
Unsquashfs will refuse to output.
3. The following major bug fixes have been made
3.1 A fragment table rounding bug has been fixed in Mksquashfs.
Previously if the number of fragments in the filesystem
were a multiple of 512, Mksquashfs would generate an
incorrect filesystem.
3.2 A rare SMP bug which occurred when simultaneously acccessing
multiply mounted Squashfs filesystems has been fixed.
4. Miscellaneous improvements/bug fixes
4.1 Kernel code stack usage has been reduced. This is to ensure
Squashfs works with 4K stacks.
4.2 Readdir (Squashfs kernel code) has been fixed to always
return 0, rather than the number of directories read. Squashfs
should now interact better with NFS.
4.3 Lseek bug in Mksquashfs when appending to larger than 4GB
filesystems fixed.
4.4 Squashfs 2.x initrds can now been mounted.
4.5 Unsquashfs exit status fixed.
4.6 New patches for linux-2.6.18 and linux-2.4.33.
3.0 15 MAR 2006 Major filesystem improvements
1. Filesystems are no longer limited to 4 GB. In
theory 2^64 or 4 exabytes is now supported.
2. Files are no longer limited to 4 GB. In theory the maximum
file size is 4 exabytes.
3. Metadata (inode table and directory tables) are no longer
restricted to 16 Mbytes.
4. Hardlinks are now suppported.
5. Nlink counts are now supported.
6. Readdir now returns '.' and '..' entries.
7. Special support for files larger than 256 MB has been added to
the Squashfs kernel code for faster read access.
8. Inode numbers are now stored within the inode rather than being
computed from inode location on disk (this is not so much an
improvement, but a change forced by the previously listed
improvements).
2.2-r2 8 SEPT 2005 Second release of 2.2, this release fixes a couple
of small bugs, a couple of small documentation
mistakes, and adds a patch for kernel 2.6.13.
1. Mksquashfs now deletes the output filesystem image file if an
error occurs whilst generating the filesystem. Previously on
error the image file was left empty or partially written.
2. Updated mksquashfs so that it doesn't allow you to generate
filesystems with block sizes smaller than 4K. Squashfs hasn't
supported block sizes less than 4K since 2.0-alpha.
3. Mksquashfs now ignores missing files/directories in sort files.
This was the original behaviour before 2.2.
4. Fixed small mistake in fs/Kconfig where the version was still
listed as 2.0.
5. Updated ACKNOWLEDGEMENTS file.
2.2 3 JUL 2005 This release has some small improvements, bug fixes
and patches for new kernels.
1. Sort routine re-worked and debugged from release 2.1. It now allows
you to give Mkisofs style sort files and checks for filenames that
don't match anything. Sort priority has also been changed to
conform to Mkisofs usage, highest priority files are now placed
at the start of the filesystem (this means they will be on the
inside of a CD or DVD).
2. New Configure options for embedded systems (memory constrained
systems). See INSTALL file for further details.
3. Directory index bug fixed where chars were treated as signed on
some architectures. A file would not be found in the rare case
that the filename started with a chracter greater than 127.
4. Bug introduced into the read_data() routine when sped up to use data
block queueing fixed. If the second or later block resulted in an
I/O error this was not checked.
5. Append bug introduced in 2.1 fixed. The code to compute the new
compressed and uncompressed directory parts after appending was
wrong.
6. Metadata block length read routine altered to not perform a
misaligned short read. This was to fix reading on an ARM7 running
uCLinux without a misaligned read interrupt handler.
7. Checkdata bug introduced in 2.1 fixed.
2.1-r2 15 DEC 2004 Code changed so it can be compiled with gcc 2.x
1. In some of the code added for release 2.1 I unknowingly used some
gcc extensions only supported by 3.x compilers. I have received
a couple of reports that the 2.1 release doesn't build on 2.x and so
people are clearly still using gcc 2.x. The code has been
rewritten to remove these extensions.
2.1 10 DEC 2004 Significantly improved directory handling plus numerous
other smaller improvements
1. Fast indexed directories implemented. These speed up directory
operations (ls, file lookup etc.) significantly for directories
larger than 8 KB.
2. All directories are now sorted in alphabetical order. This again
speeds up directory operations, and in some cases it also results in
a small compression improvement (greater data similarity between
files with alphabetically similar names).
3. Maximum directory size increased from 512 KB to 128 MB.
4. Duplicate fragment checking and appending optimised in mksquashfs,
depending on filesystem, this is now up to 25% faster.
5. Mksquashfs help information reformatted and reorganised.
6. The Squashfs version and release date is now printed at kernel
boot-time or module insertion. This addition will hopefully help
to reduce the growing problem where the Squashfs version supported
by a kernel is unknown and the kernel source is unavailable.
7. New PERFORMANCE.README file.
8. New -2.0 mksquashfs option.
9. CHANGES file reorganised.
10. README file reorganised, clarified and updated to include the 2.0
mksquashfs options.
11. New patch for Linux 2.6.9.
12. New patch for Linux 2.4.28.
2.0r2 29 AUG 2004 Workaround for kernel bug in kernels 2.6.8 and newer
added
1. New patch for kernel 2.6.8.1. This includes a workaround for a
kernel bug introduced in 2.6.7bk14, which is present in all later
versions of the kernel.
If you're using a 2.6.8 kernel or later then you must use this
2.6.8.1 patch. If you've experienced hangs or oopses using Squashfs
with a 2.6.8 or later kernel then you've hit this bug, and this
patch will fix it.
It is worth mentioning that this kernel bug potentially affects
other filesystems. If you receive odd results with other
filesystems you may be experiencing this bug with that filesystem.
I submitted a patch but this has not yet gone into the
kernel, hopefully the bug will be fixed in later kernels.
2.0 13 JULY 2004 A couple of new options, and some bug fixes
1. New mksquashfs -all-root, -root-owned, -force-uid, and -force-gid
options. These allow the uids/gids of files in the generated
filesystem to be specified, overriding the uids/gids in the
source filesystem.
2. Initrds are now supported for kernels 2.6.x.
3. amd64 bug fixes. If you use an amd64, please read the README-AMD64
file.
4. Check-data and gid bug fixes. With 2.0-alpha when mounting 1.x
filesystems in certain cases file gids were corrupted.
5. New patch for Linux 2.6.7.
2.0-ALPHA 21 MAY 2004 Filesystem changes and compression improvements
1. Squashfs 2.0 has added the concept of fragment blocks.
Files smaller than the file block size and optionally the
remainder of files that do not fit fully into a block (i.e. the
last 32K in a 96K file) are packed into shared fragments and
compressed together. This achieves on average 5 - 20% better
compression than Squashfs 1.x.
2. The maximum block size has been increased to 64K (in the ALPHA
version of Squashfs 2.0).
3. The maximum number of UIDs has been increased to 256 (from 48 in
1.x).
4. The maximum number of GIDs has been increased to 256 (from 15 in
1.x).
5. Removal of sleep_on() function call in 2.6.x patch, to allow Squashfs
to work on the Fedora rc2 kernel.
6. Numerous small bug fixes have been made.
1.3r3 18 JAN 2004 Third release of 1.3, this adds a new mksquashfs option,
some bug fixes, and extra patches for new kernels
1. New mksquashfs -ef exclude option. This option reads the exclude
dirs/files from an exclude file, one exclude dir/file per line. This
avoids the command line size limit when using the -e exclude option,
2. When appending to existing filesystems, if mksquashfs experiences a
fatal error (e.g. out of space when adding to the destination), the
original filesystem is restored,
3. Mksquashfs now builds standalone, without the kernel needing to be
patched.
4. Bug fix in the kernel squashfs filesystem, where the pages being
filled were not kmapped. This seems to only have caused problems
on an Apple G5,
5. New patch for Linux 2.4.24,
6. New patch for Linux 2.6.1, this replaces the patch for 2.6.0-test7.
1.3r2 14 OCT 2003 Second release of 1.3, bug fixes and extra patches for
new kernels
1. Bug fix in routine that adds files to the filesystem being
generated in mksquashfs. This bug was introduced in 1.3
(not enough testing...) when I rewrote it to handle files larger
than available memory. This bug caused a SEGV, so if you've ever
got that, it is now fixed,
2. Long running bug where ls -s and du reported wrong block size
fixed. I'm pretty sure this used to work many kernel versions ago
(2.4.7) but it broke somewhere along the line since then,
3. New patch for Linux 2.4.22,
4. New patch for 2.6.0-test7, this replaces the patch for 2.6.0-test1.
1.3 29 JUL 2003 FIFO/Socket support added plus optimisations and
improvements
1. FIFOs and Socket inodes are now supported,
2. Mksquashfs can now compress files larger than available
memory,
3. File duplicate check routine optimised,
4. Exit codes fixed in Mksquashfs,
5. Patch for Linux 2.4.21,
6. Patch for Linux 2.6.0-test1. Hopefully, this will work for
the next few releases of 2.6.0-testx, otherwise, I'll be
releasing a lot of updates to the 2.6.0 patch...
1.2 13 MAR 2003 Append feature and new mksquashfs options added
Mksquashfs can now add to existing squashfs filesystems. Three extra
options "-noappend", "-keep-as-directory", and "root-becomes"
have been added.
The append option with file duplicate detection, means squashfs can be
used as a simple versioning archiving filesystem. A squashfs
filesystem can be created with for example the linux-2.4.19 source.
Appending the linux-2.4.20 source will create a filesystem with the
two source trees, but only the changed files will take extra room,
the unchanged files will be detected as duplicates.
See the README file for usage changes.
1.1b 16 JAN 2003 Bug fix release
Fixed readpage deadlock bug. This was a rare deadlock bug that
happened when pushing pages into the page cache when using greater
than 4K blocks. I never got this bug when I tested the filesystem,
but two people emailed me on the same day about the problem!
I fixed it by using a page cache function that wasn't there when
I originally did the work, which was nice :-)
1.1 8 JAN 2003 Added features
1. Kernel squashfs can now mount different byte order filesystems.
2. Additional features added to mksquashfs. Mksquashfs now supports
exclude files and multiple source files/directories can be
specified. A nopad option has also been added, which
informs mksquashfs not to pad filesystems to a multiple of 4K.
See README for mksquashfs usage changes.
3. Greater than 2GB filesystems bug fix. Filesystems greater than 2GB
can now be created.
1.0c 14 NOV 2002 Bug fix release
Fixed bugs with initrds and device nodes
1.0 23 OCT 2002 Initial release
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License.
INSTALLING SQUASHFS
1. Kernel support
-----------------
This release is for 2.6.29 and newer kernels. Kernel patching is not necessary.
Extended attribute support requires 2.6.35 or newer kernels. But
file systems with extended attributes can be mounted on 2.6.29 and
newer kernels (the extended attributes will be ignored with a warning).
LZO compression support requires 2.6.36 or newer kernels.
XZ compression support requires 2.6.38 or newer kernels.
LZ4 compression support requires 3.11 or newer kernels.
ZSTD compression support requires 4.14 or newer kernels.
2. Building squashfs tools
--------------------------
The squashfs-tools directory contains the source code for the Mksquashfs and
Unsquashfs programs. These can be compiled by typing make (or sudo make
install to install in /usr/local/bin) within that directory.
2.1 Compressors supported
By default the Makefile is configured to build Mksquashfs and Unsquashfs
with GZIP suppport. Read the Makefile in squashfs-tools for instructions on
building LZO, LZ4, XZ and ZSTD compression support.
2.2 Extended attribute support
By default the Makefile is configured to build Mksquashfs and Unsquashfs
with extended attribute support. Read the Makefile in squashfs-tools for
instructions on how to disable extended attribute support, if not supported
by your distribution/C library, or if it is not needed.
2.3 Reproducible builds
By default the Makefile is configured to build Mksquashfs with reproducible
output as default. Read the Makefile in squashfs-tools for instructions
on how to disable reproducible output as default if desired.
GitHub
This is now the main development repository for Squashfs-Tools.
There are older repositories on Sourceforge and kernel.org. These
are currently out of date, but should be updated in the near future.
The kernel directories are obsolete, all kernel code is now in
mainline at www.kernel.org.
kernel-2.4: If you still need Squashfs support in the 2.4 kernel
then use the squashfs 3.2-r2 release. This is the last release
that has a Mksquashfs which generates filesystems mountable with
Squashfs patched 2.4 kernels. This release has patches for 2.4
kernels (but they have not been updated since the 3.1 release).
SQUASHFS 4.4 - A squashed read-only filesystem for Linux
Copyright 2002-2019 Phillip Lougher <phillip@squashfs.org.uk>
Released under the GPL licence (version 2 or later).
Welcome to Squashfs 4.4. This is the first release in over 5 years, and
there are substantial improvements: reproducible builds, new compressors,
CVE fixes, security hardening and new options for Mksquashfs/Unsquashfs.
Please see the INSTALL file for instructions on installing the tools,
and the USAGE file for documentation on how to use the tools.
Summary of changes (and sections below)
---------------------------------------
1. Mksquashfs now generates reproducible images by default. Mkfs time and
file timestamps can also be specified.
2. Support for the Zstandard (ZSTD) compression algorithm has been added.
3. Pseudo files now support symbolic links.
4. CVE-2015-4645 and CVE-2015-4646 have been fixed.
5. Unsquashfs has been further hardened against corrupted filestems.
6. Unsquashfs is now more strict about error handling.
7. Miscellaneous new options and major bug fixes for Mksquashfs.
8. Miscellaneous new options and major bug fixes for Unsquashfs.
9. Squashfs-tools 4.4 is compatible with all earlier 4.x filesystems
and releases.
1. Introducing reproducible builds
----------------------------------
Ever since Mksquashfs was parallelised back in 2006, there
has been a certain randomness in how fragments and multi-block
files are ordered in the output filesystem even if the input
remains the same.
This is because the multiple parallel threads can be scheduled
differently between Mksquashfs runs. For example, the thread
given fragment 10 to compress may finish before the thread
given fragment 9 to compress on one run (writing fragment 10
to the output filesystem before fragment 9), but, on the next
run it could be vice-versa. There are many different scheduling
scenarios here, all of which can have a knock on effect causing
different scheduling and ordering later in the filesystem too.
Mkquashfs doesn't care about the ordering of fragments and
multi-block files within the filesystem, as this does not
affect the correctness of the filesystem.
In fact not caring about the ordering, as it doesn't matter, allows
Mksquashfs to run as fast as possible, maximising CPU and I/O
performance.
But, in the last couple of years, Squashfs has become used in
scenarios (cloud etc) where this randomness is causing problems.
Specifically this appears to be where downloaders, installers etc.
try to work out the differences between Squashfs filesystem
updates to minimise the amount of data that needs to transferred
to update an image.
Additionally, in the last couple of years has arisen the notion
of reproducible builds, that is the same source and build
environment etc should be able to (re-)generate identical
output. This is usually for verification and security, allowing
binaries/distributions to be checked for malicious activity.
See https://reproducible-builds.org/ for more information.
Mksquashfs now generates reproducible images by default.
Images generated by Mksquashfs will be ordered identically to
previous runs if the same input has been supplied, and the
same options used.
1.1.1 Dealing with timestamps
Timestamps embedded in the filesystem will stiil cause differences.
Each new run of Mksquashfs will produce a different mkfs (make filesystem)
timestamp in the super-block. Moreover if any file timestamps have changed
(even if the content hasn't), this will produce a difference.
To prevent timestamps from producing differences, the following
new Mksquashfs options have been added.
1.1.2 -mkfs-time <time>
This option takes a positive time value (which is the number
of seconds since the epoch of 1970-01-01 00:00:00 UTC), and sets
the file system timestamp to that.
Squashfs uses an unsigned 32-bit integer to store time, and the
time given should be in that range.
Obviously you can use the date command to convert dates into
this value, i.e.
% mksquashfs source source.sqsh -mkfs-time $(date +%s -d "Jan 1 2019 19:00")
1.1.3 -all-time <time>
This option takes a positive time value (which is the number
of seconds since the epoch of 1970-01-01 00:00:00 UTC), and sets
the timestamp on all files to that (but not the mkfs time).
1.1.4 environment variable SOURCE_DATE_EPOCH
As an alternative to the above command line options, you can
set the environment variable SOURCE_DATE_EPOCH to a time value.
This value will be used to set the mkfs time. Also any
file timestamps which are after SOURCE_DATE_EPOCH will be
clamped to SOURCE_DATE_EPOCH.
See https://reproducible-builds.org/docs/source-date-epoch/
for more information.
Note: both SOURCE_DATE_EPOCH and the command line options cannot
be used at the same time. They are different ways to do the same thing,
and both have FORCE sematics which mean they can't be over-ridden
elsewhere (otherwise it would defeat the purpose).
1.1.5 -not-reproducible
This option tells Mksquashfs that the files do not have to be
strictly ordered. This will make Mksquashfs behave like version 4.3.
2. Zstandard (ZSTD) compression added
-------------------------------------
This is named zstd. It supports the following
compression options.
zstd
-Xcompression-level <compression-level>
<compression-level> should be 1 .. 22 (default 15)
3. Pseudo file symbolic link support added
------------------------------------------
Pseudo definition
Filename s mode uid gid symlink
uid and gid can be either specified as a decimal number, or by name.
Note mode is ignored, as symlinks always have "rwxrwxrwx" permissions.
For example:
symlink s 0 root root example
creates a symlink "symlink" to file "example" with root uid/gid.
4. CVE-2015-2015-4645 and CVE-2015-4646
---------------------------------------
These CVEs were raised due to Unsquashfs having variable overflow and
stack overflow in a number of vulnerable functions.
All instances of variable overflow and stack overflow have been
removed.
5. Unsquashfs hardened against corrupted filestems
--------------------------------------------------
The filesystem super-block values and filesystem metadata tables
are further sanity checked. More importantly, these values are now
checked for validity against other values in the metadata tables, and
these values must match.
6. Unsquashfs is now more strict about error handling
-----------------------------------------------------
Unsquashfs splits errors into two categories: fatal errors and non-fatal
errors. In this release a significant number of errors that were previously
non-fatal have been hardened to fatal.
Fatal errors are those which cause Unsquashfs to abort instantly.
These are generally due to failure to read the filesystem (corruption),
and/or failure to write files to the output filesystem, due to I/O error
or out of space. Generally anything which is unexpected is a fatal error.
Non-fatal errors are generally where support is lacking in the
output filesystem, and it can be considered to be an expected failure.
This includes the inability to write extended attributes (xattrs) to
a filesystem that doesn't support them, the inability to create files on
filesystem that doesn't support them (i.e. symbolic links on VFAT), and the
inability to execute privileged operations as a user-process.
The user may well know the filesystem cannot support certain operations
and would prefer Unsquashfs to ignore then without aborting.
Two new options have been added:
6.1. -ignore-errors
This makes Unsquashfs behave like previous versions, and treats more
errors as non-fatal.
6.2. -strict-errors
This makes Unsquashfs treat every error as fatal, and it will abort
instantly.
7. Miscellaneous new options and major bug fixes for Mksquashfs
---------------------------------------------------------------
7.1. -root-mode <mode>
This sets the permissions of the root directory to the octal <mode>.
This is mostly intended for when multiple sources are specified on
the command line. In this instance Mksquashfs produces a dummy top level
directory with permissions 0777 (rwxrwxrwx). This option allows the
permissions to be changed. But the option can also be used when a single
source is specified.
7.2. -quiet
This suppresses all output from Mksquashfs, except the progress bar.
The progress bar can disabled with -no-progress to produce completely
silent output.
This new option is useful for scripts.
7.3. -noId
This is similar to the pre-existing -noI option, except, it specifies that
only the Id table (uids and gids) should be uncompressed. This option was
added to enable a use-case where uids and gids need to be updated after
filesystem generation.
7.4. -offset <offset>
This option skips <offset> bytes at the beginning of the output filesystem.
Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or
Gbytes respectively.
7.5. Update lz4 wrapper to use new functions introduced in 1.7.0
7.5. Bug fix, don't allow "/" pseudo filenames
7.6. Bug fix, allow quoting of pseudo files, to better handle filenames with
spaces
7.7. Fix compilation with glibc 2.25+
8. Miscellaneous new options and major bug fixes for Unsquashfs
---------------------------------------------------------------
8.1. -lln[umeric]
This is similar to the "-lls" option but displays uids and gids numerically.
8.2. -lc option
This is similar to the "-ls" option except it only displays files and empty
directories.
8.3. -llc option
As "-llc" option but displays file attributes.
8.4. -offset <offset>
This option skips <offset> bytes at the beginning of the input filesystem.
Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or
Gbytes respectively.
8.5. -quiet
This suppresses all output from Unsquashfs, except the progress bar.
The progress bar can disabled with -no-progress to produce completely
silent output.
This new option is useful for scripts.
8.6. -UTC
This option makes Unsquashfs display all times in the UTC time zone rather
than using the default local time zone.
8.7. Update lz4 wrapper to use new functions introduced in 1.7.0
8.8. Bug fix, fatal and non-fatal errors now set the exit code to 1
8.9. Bug fix, fix time setting for symlinks
8.10. Bug fix, try to set sticky-bit when running as a user process
8.11. Fix compilation with glibc 2.25+
9. Compatiblity
---------------
Mksquashfs 4.4 generates 4.0 filesystems. These filesystems are fully
compatible/interchangable with filesystems generated by Mksquashfs 4.x and are
mountable on 2.6.29 and later kernels.
Help sponsor Squashfs development!
Maintaining and improving Squashfs is a lot of work, but Squashfs is one of
the only widely used Linux file systems that has no company backing. Squashfs
development is funded soley by the author, partially supported by donations
from companies and individuals that want to improve Squashfs for themselves
and others.
There's lots of exciting new improvements to Squashfs in the pipeline, and
if your company is a serious user of Squashfs, please consider accelerating
development of Squashfs by donating.
Donatations can be made from the Squashfs sourceforge homepage, or if you
prefer by contacting the author privately.
* Obsolete *
This file is now largely obsolete, considering
it refers to Squashfs 2.1 (which was released in 2006)
and older releases.
GENERAL INFORMATION ON PERFORMANCE TESTS
----------------------------------------
The following performance tests were based on two file sets: the
liveCD filesystem from the Ubuntu liveCD (Warty release), and the
liveCD filesystem from the Damn Small Linux liveCD (release 0.8.4).
The Ubuntu liveCD filesystem was used to test filesystem performance
from CDROM and hard disk for Zisofs, Cloop, Squashfs 2.0 and Squashfs2.1.
CRAMFS filesystem performance could not be tested for this filesystem
bacause it exceeds the maximum supported size of CRAMFS. To test
CRAMFS performance against Squashfs, the liveCD filesystem from
Damn Small Linux was used.
NOTE: the usual warnings apply to these results, they are provided for
illustrative purposes only, and due to different hardware and/or file data, you
may obtain different results. As such the results are provided "as is" without
any warranty (either express or implied) and you assume all risks as to their
quality and accuracy.
1. Ubuntu liveCD performance tests
ext3 uncompressed size 1.4 GB
Zisofs compressed size 589.81 MB
Cloop compressed size 471.89 MB
Squashfs2.0 compressed size 448.58 MB
Squashfs2.1 compressed size 448.58 MB
1.1 Performance tests from CDROM
1.1.1 Directory Lookup performance
Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem
mounted from CDROM
Zisofs 49.88 seconds (User 2.60 secs, Sys 11.19 secs)
Cloop 20.80 seconds (User 2.71 secs, Sys 13.50 secs)
Squashfs2.0 16.56 seconds (User 2.42 secs, Sys 10.37 secs)
Squashfs2.1 10.14 seconds (User 2.48 secs, Sys 4.44 secs)
1.1.2 Sequential I/O performance
Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted
from CDROM
Zisofs 27 minutes 28.54 seconds (User 3.00 secs, Sys 1 min 4.80 secs)
Cloop 5 minutes 55.72 seconds (User 2.90 secs, Sys 3 min 37.90 secs)
Squashfs2.0 5 minutes 20.87 seconds (User 2.33 secs, Sys 56.98 secs)
Squashfs2.1 5 minutes 15.46 seconds (user 2.28 secs, Sys 51.12 secs)
1.1.3 Random I/O performance
Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort
-g | awk '{ printf $2 }' > /tmp/sort
Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null"
on filesystem mounted from CDROM
Zisofs 101 minutes 29.65 seconds (User 5.33 secs, Sys 1 min 17.20 secs)
Cloop 35 minutes 27.51 seconds (user 5.93 secs, Sys 4 mins 30.23 secs)
Squashfs2.0 21 minutes 53.05 seconds (user 5.71 secs, Sys 2 mins 36.59 secs)
Squashfs2.1 21 minutes 46.99 seconds (User 5.80 secs, Sys 2 mins 31.88 secs)
1.2 Performance tests from Hard disk
1.2.1 Directory Lookup performance
Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem
mounted from Hard disk
Zisofs 17.29 seconds (User 2.62 secs, Sys 11.08 secs)
Cloop 16.46 seconds (User 2.63 secs, Sys 13.41 secs)
Squashfs2.0 13.75 seconds (User 2.44 secs, Sys 11.00 secs)
Squashfs2.1 6.94 seconds (User 2.44 secs, Sys 4.48 secs)
1.2.2 Sequential I/O performance
Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted
from Hard disk
Zisofs 1 minute 21.47 seconds (User 2.73 secs, Sys 54.44 secs)
Cloop 1 minute 34.06 seconds (user 2.85 secs, Sys 1 min 12.13 secs)
Squashfs2.0 1 minute 21.22 seconds (User 2.42 secs, Sys 56.21 secs)
Squashfs2.1 1 minute 15.46 seconds (User 2.36 secs, Sys 49.78 secs)
1.2.3 Random I/O performance
Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort
-g | awk '{ printf $2 }' > /tmp/sort
Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null"
on filesystem mounted from Hard disk
Zisofs 11 minutes 13.64 seconds (User 5.08 secs, Sys 52.62 secs)
Cloop 5 minutes 37.93 seconds (user 6 secs, Sys 2 mins 22.38 secs)
Squashfs2.0 5 minutes 7.11 seconds (user 5.63 secs, Sys 2 mins 35.23 secs)
Squashfs2.1 5 minutes 1.87 seconds (User 5.71 secs, Sys 2 mins 29.98 secs)
2. Damn Small Linux liveCD performance tests
ext3 uncompressed size 126 MB
CRAMFS compressed size 52.19 MB
Squashfs2.0 compressed size 46.52 MB
Squashfs2.1 compressed size 46.52 MB
2.1 Performance tests from CDROM
2.1.1 Directory Lookup performance
Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem
mounted from CDROM
CRAMFS 10.85 seconds (User 0.39 secs, Sys 0.98 secs)
Squashfs2.0 2.97 seconds (User 0.36 secs, Sys 2.15 secs)
Squashfs2.1 2.43 seconds (User 0.40 secs, Sys 1.42 secs)
2.1.2 Sequential I/O performance
Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted
from CDROM
CRAMFS 55.38 seconds (User 0.34 secs, Sys 6.98 secs)
Squashfs2.0 35.99 seconds (User 0.30 secs, Sys 6.35 secs)
Squashfs2.1 33.83 seconds (User 0.26 secs, Sys 5.56 secs)
2.1.3 Random I/O performance
Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort
-g | awk '{ printf $2 }' > /tmp/sort
Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null"
on filesystem mounted from CDROM
CRAMFS 3 minutes 1.68 seconds (User 0.54 secs, Sys 9.51 secs)
Squashfs2.0 1 minute 39.45 seconds (User 0.57 secs, Sys 13.14 secs)
Squashfs2.1 1 minute 38.41 seconds (User 0.58 secs, Sys 13.08 secs)
2.2 Performance tests from Hard disk
2.2.1 Directory Lookup performance
Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem
mounted from Hard disk
CRAMFS 1.77 seconds (User 0.53 secs, Sys 1.21 secs)
Squashfs2.0 2.67 seconds (User 0.41 secs, Sys 2.25 secs)
Squashfs2.1 1.87 seconds (User 0.41 secs, Sys 1.46 secs)
2.2.2 Sequential I/O performance
Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted
from Hard disk
CRAMFS 6.80 seconds (User 0.36 secs, Sys 6.02 secs)
Squashfs2.0 7.23 seconds (User 0.29 secs, Sys 6.62 secs)
Squashfs2.1 6.53 seconds (User 0.31 secs, Sys 5.82 secs)
2.2.3 Random I/O performance
Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort
-g | awk '{ printf $2 }' > /tmp/sort
Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null"
on filesystem mounted from Hard disk
CRAMFS 28.55 seconds (User 0.49 secs, Sys 6.49 secs)
Squashfs2.0 25.44 seconds (User 0.58 secs, Sys 13.17 secs)
Squashfs2.1 24.72 seconds (User 0.56 secs, Sys 13.15 secs)
NOTE: This the original README for version 2.0. It is retained as it
contains information about the fragment design. A description of the new 2.0
mksquashfs options has been added to the main README file, and that
file should now be consulted for these.
SQUASHFS 2.0 - A squashed read-only filesystem for Linux
Copyright 2004 Phillip Lougher (plougher@users.sourceforge.net)
Released under the GPL licence (version 2 or later).
Welcome to the final release of Squashfs version 2.0! A lot of changes to the
filesystem have been made under the bonnet (hood). Squashfs 2.0 uses fragment
blocks and larger blocks (64K) to improve compression ratio by about 5 - 20%
over Squashfs 1.0 depending on the files being compressed. Using fragment
blocks allows Squashfs 2.0 to achieve better compression than cloop and similar
compression to tgz files while retaining the I/O efficiency of a compressed
filesystem.
Detailed changes:
1. Squashfs 2.0 has added the concept of fragment blocks (see later discussion).
Files smaller than the file block size (64K in Squashfs 2.0) and optionally
the remainder of files that do not fit fully into a block (i.e. the last 32K
in a 96K file) are packed into shared fragments and compressed together.
This achieves on average 5 - 20% better compression than Squashfs 1.x.
2. The maximum block size has been increased to 64K.
3. The maximum number of UIDs has been increased to 256 (from 48 in 1.x).
4. The maximum number of GIDs has been increased to 256 (from 15 in 1.x).
5. New mksquashfs -all-root, -root-owned, -force-uid, and -force-gid
options. These allow the uids/gids of files in the generated
filesystem to be specified, overriding the uids/gids in the
source filesystem.
6. Initrds are now supported for kernels 2.6.x.
7. Removal of sleep_on() function call in 2.6.x patch, to allow Squashfs
to work on the Fedora rc2 kernel.
8. AMD64, check-data and gid bug fixes.
9. Numerous small bug fixes have been made.
10. New patch for Linux 2.6.7.
New Squashfs 2.0 options
------------------------
-noF or -noFragmentCompression
Do not compress the fragments. Added for compatibility with noI and
noD, probably not that useful.
-no-fragments
Do not use fragment blocks, and rather generate a filesystem
similar to a Squashfs 1.x filesystem. It will of course still
be a Squashfs 2.0 filesystem but without fragments, and so
it won't be mountable on a Squashfs 1.x system.
-always-use-fragments
By default only small files less than the block size are packed into
fragment blocks. The ends of files which do not fit fully into a block,
are NOT by default packed into fragments. To illustrate this, a
100K file has an initial 64K block and a 36K remainder. This
36K remainder is not packed into a fragment by default. This is
because to do so leads to a 10 - 20% drop in sequential I/O
performance, as a disk head seek is needed to seek to the initial
file data and another disk seek is need to seek to the fragment
block.
Specify this option if you want file remainders to be packed into
fragment blocks. Doing so may increase the compression obtained
BUT at the expense of I/O speed.
-no-duplicates
Do not detect duplicate files.
-all-root
-root-owned
These options (both do exactly the same thing), force all file
uids/gids in the generated Squashfs filesystem to be root.
This allows root owned filesystems to be built without root access
on the host machine.
-force-uid uid
This option forces all files in the generated Squashfs filesystem to
be owned by the specified uid. The uid can be specified either by
name (i.e. "root") or by number.
-force-gid gid
This option forces all files in the generated Squashfs filesystem to
be group owned by the specified gid. The gid can be specified either by
name (i.e. "root") or by number.
Compression improvements example
--------------------------------
The following is the compression results obtained compressing the 2.6.6
linux kernel source using CRAMFS, Cloop (with iso filesystem), Squashfs 1.3 and
Squashfs 2.0 (results generated using big-endian filesystems).
In decreasing order of size:
CRAMFS 62791680 bytes (59.9M)
Squashfs 1.x 51351552 bytes (48.9M)
Cloop 46118681 bytes (44.0M)
Squashfs 2.0 45604854 bytes (43.5M)
The Squashfs 1.x filesystem is 12.6% larger than the new 2.0 filesystem.
The cloop filesystem is 1.1% larger than the Squashfs 2.0 filesystem.
Fragment blocks in Squashfs 2.0
-------------------------------
Squashfs like all other compressed filesystems compresses files individually
on a block by block basis. This is performed to allow mounting and
de-compression of files on a block by block basis without requiring the entire
filesystem to be decompressed. This is in contrast to data-based compression
schemes which compress without understanding the underlying filesystem (i.e.
cloop and tgz files) and which, therefore, do not compress files individually.
Each approach has advantages and disadvantages, data-based systems have better
compression because compression is always performed at the maximum block size
(64K in cloop) irrespective of the size of each file (which could be less than
the block size). Compressed filesystems tend to be faster at I/O because
they understand the filesystem and therefore employ better caching stategies
and read less un-needed data from the filesystem.
Fragment blocks in Squashfs 2.0 solves this problem by packing files (and
optionally the ends of files) which are smaller than the block size into
shared blocks, which are compressed together. For example five files each of
10K will be packed into one shared fragment of 50K and compressed together,
rather than being compressed in five 10K blocks.
This scheme produces a hybrid filesystem, retaining the I/O efficiency
of a compressed filesystem, while obtaining the compression efficiency
of data-based schemes by compressing small files together.
Squashfs 1.x and Squashfs 2.0 compatibility
-------------------------------------------
Appending to Squashfs 1.x filesystems is not supported. If you wish to append
to 1.x filesystems, then either use the original mksquashfs, or convert them
to Squashfs 2.0 by mounting the filesystem and running the 2.0 mksquashfs
on the mounted filesystem.
Mounting Squashfs 1.x filesystems IS supported by the 2.0 kernel patch.
Information for amd64 users
---------------------------
All previous releases of Squashfs (2.0-alpha and older) generate incorrect
filesystems on amd64 machines. These filesystems work correctly on amd64
machines, but cannot be mounted on non-amd64 machines. Likewise, filesystems
generated on non amd64 machines could not be mounted on amd64 machines.
This bug was caused by the different size of the "time_t" definition used in
SquashFS filesystem structures.
This bug is now fixed in this release. However, all amd64 filesystems
generated by previous releases will not be mountable on amd64 machines
with this release. If you have pre-existing amd64 generated filesystems,
it is important that you recreate the filesystem. This can be performed
by mounting the filesystem using a kernel with the original patch
(i.e. a 2.0-alpha or older patch) and running the SquashFS 2.0
(i.e. this release) mksquashfs tool to create a new SquashFS filesystem.
SQUASHFS 2.1 - A squashed read-only filesystem for Linux
Copyright 2004 Phillip Lougher (plougher@users.sourceforge.net)
Released under the GPL licence (version 2 or later).
Welcome to Squashfs version 2.1-r2. Squashfs 2.1 introduces indexed
directories which considerably speed up directory lookup (ls, find etc.) for
directories which are greater than 8K in size. All directories are now also
sorted alphabetically which further speeds up directory lookup. Many smaller
improvements have also been made to this release, please see the CHANGES file
entry for detailed changes.
1. DIRECTORY SPEED IMPROVEMENT EXAMPLES
---------------------------------------
To give an indication of the directory speed improvements a number of test
results are shown here. There is in addition a new PERFORMANCE.README file
which gives details of I/O and lookup performance for Squashfs 2.1 against
the Zisofs, Cloop and CRAMFS filesystems.
example 1:
Filesystems generated from a single directory of 72,784 files (2.6 MB
directory size). Each file is 10 bytes in size (the test is directory
lookup and so the file size isn't an issue). The ext3 uncompressed
directory size is 288 MB (presumably because of one file per block).
Zisofs compressed size 153.50 MB
Cloop (isofs) compressed size 1.74 MB
Squashfs2.1 compressed size 612 KB (0.60 MB)
Time taken to perform "ls -lR --color=always | cat > /dev/null" on
filesystems mounted on hard disk.
Zisofs 35 minutes 7.895 seconds (User 7.868 secs, Sys 34 mins 5.621 secs)
Cloop 35 minutes 12.765 seconds (User 7.771 secs, Sys 34 mins 3.869 secs)
Squashfs2.1 19 seconds (User 5.119 secs, Sys 14.547 secs)
example 2:
Filesystems were generated from the Ubuntu Warty livecd (original uncompressed
size on ext3 is 1.4 GB).
Zisofs compressed size 589.81 MB
Cloop (isofs) compressed size 471.19 MB
Squashfs2.0 compressed size 448.58 MB
Squashfs2.1 compressed size 448.58 MB
Time taken to perform "ls -lR --color=always | cat > /dev/null" on
filesystems mounted on hard disk.
Zisofs 49.875 seconds (User time 2.589 secs, Sys 11.194 secs)
Cloop 20.797 seconds (User time 2.706 secs, Sys 13.496 secs)
Squashfs2.0 16.556 seconds (User time 2.424 secs, Sys 10.371 secs)
Squashfs2.1 10.143 seconds (User time 2.475 secs, Sys 4.440 secs)
NOTE: the usual warnings apply to these results, they are provided for
illustrative purposes only, and due to different hardware and/or file data, you
may obtain different results. As such the results are provided "as is" without
any warranty (either express or implied) and you assume all risks as to their
quality and accuracy.
2. NEW MKSQUASHFS OPTIONS
-------------------------
There is only one extra option "-2.0". This tells mksquashfs to generate
a filesystem which is mountable with Squashfs version 2.0.
3. APPENDING AND MOUNTING SQUASHFS 2.0 FILESYSTEMS
--------------------------------------------------
Mounting 2.0 filesystems is supported by Squashfs 2.1. In addition
mksquashfs v2.1 can append to 2.0 filesystems, although the generated
filesystem will still be a 2.0 filesystem.
4. DONATIONS
------------
If you find Squashfs useful then please consider making a donation,
particularly if you use Squashfs in a commercial product. Please consider
giving something back especially if you're making money from it.
Off the Squashfs subject somewhat I'm currently looking for another
job doing Linux kernel or filesystems work. If you know of any such
work that can be performed from the UK then please get in touch. Thanks.
SQUASHFS 3.0 - A squashed read-only filesystem for Linux
Copyright 2002-2006 Phillip Lougher <phillip@lougher.org.uk>
Released under the GPL licence (version 2 or later).
Welcome to the first release of Squashfs version 3.0. Squashfs 3.0 has the
the following improvements to 2.x.
1. Filesystems are no longer limited to 4 GB. In
theory 2^64 or 4 exabytes is now supported.
2. Files are no longer limited to 4 GB. In theory the maximum
file size is 4 exabytes.
3. Metadata (inode table and directory tables) are no longer
restricted to 16 Mbytes.
4. Hardlinks are now suppported.
5. Nlink counts are now supported.
6. Readdir now returns '.' and '..' entries.
7. Special support for files larger than 256 MB has been added to
the Squashfs kernel code for faster read access.
8. Inode numbers are now stored within the inode rather than being
computed from inode location on disk (this is not so much an
improvement, but a change forced by the previously listed
improvements).
There is a new Unsquashfs utility (in squashfs-tools) than can be used to
decompress a filesystem without mounting it.
Squashfs 3.0 supports 2.x filesystems. Support for 1.x filesystems
will be added in the future.
1. UNSQUASHFS
-------------
Unsquashfs has the following options:
SYNTAX: unsquashfs [-ls | -dest] filesystem
-version print version, licence and copyright information
-info print files as they are unsquashed
-ls list filesystem only
-dest <pathname> unsquash to <pathname>, default "squashfs-root"
The "-ls" option can be used to list the contents of a filesystem without
decompressing the filesystem data itself.
The "-info" option forces Unsquashfs to print each file as it is decompressed.
The "-dest" option specifies the directory that is used to decompress
the filesystem data. If this option is not given then the filesystem is
decompressed to the directory "squashfs-root" in the current working directory.
Unsquashfs can decompress 3.0 filesystems. Support for 2.x and 1.x
filesystems will be added in the future.
SQUASHFS 3.1 - A squashed read-only filesystem for Linux
Copyright 2002-2006 Phillip Lougher <phillip@lougher.org.uk>
Released under the GPL licence (version 2 or later).
Welcome to Squashfs version 3.1-r2. Squashfs 3.1 has major improvements to
the Squashfs tools (Mksquashfs and Unsquashfs), some major bug fixes, new
kernel patches, and various other smaller improvements and bug fixes.
Please see the CHANGES file for a detailed list.
1. MKSQUASHFS
-------------
Mksquashfs has been rewritten and it is now multi-threaded. It offers
the following improvements:
1. Parallel compression. By default as many compression and fragment
compression threads are created as there are available processors.
This significantly speeds up performance on SMP systems.
2. File input and filesystem output is peformed in parallel on separate
threads to maximise I/O performance. Even on single processor systems
this speeds up performance by at least 10%.
3. Appending has been significantly improved, and files within the
filesystem being appended to are no longer scanned and checksummed. This
significantly improves append time for large filesystems.
4. File duplicate checking has been optimised, and split into two separate
phases. Only files which are considered possible duplicates after the
first phase are checksummed and cached in memory.
5. The use of swap memory was found to significantly impact performance. The
amount of memory used to cache the file is now a command line option, by default
this is 512 Mbytes.
1.1 NEW COMMAND LINE OPTIONS
----------------------------
The new Mksquashfs program has a couple of extra command line options
which can be used to control the new features:
-processors <processors>
This specifies the number of processors used by Mksquashfs.
By default this is the number of available processors.
-read_queue <size in Mbytes>
This specifies the size of the file input queue used by the reader thread.
This defaults to 64 Mbytes.
-write_queue <size in Mbytes>
This specifies the size of the filesystem output queue used by the
writer thread. It also specifies the maximum cache used in file
duplicate detection (the output queue is shared between these tasks).
This defaults to 512 Mbytes.
1.2 PERFORMANCE RESULTS
-----------------------
The following results give an indication of the speed improvements. Two
example filesystems were tested, a liveCD filesystem (about 1.8 Gbytes
uncompressed), and my home directory consisting largely of text files
(about 1.3 Gbytes uncompressed). Tests were run on a single core
and a dual core system.
Dual Core (AMDx2 3800+) system:
Source directories on ext3.
LiveCD, old mksquashfs:
real 11m48.401s
user 9m27.056s
sys 0m15.281s
LiveCD, new par_mksquashfs:
real 4m8.736s
user 7m11.771s
sys 0m27.749s
"Home", old mksquashfs:
real 4m34.360s
user 3m54.007s
sys 0m32.155s
"Home", new par_mksquashfs:
real 1m27.381s
user 2m7.304s
sys 0m17.234s
Single Core PowerBook (PowerPC G4 1.5 GHz Ubuntu Linux)
Source directories on ext3.
LiveCD, old mksquashs:
real 11m38.472s
user 9m6.137s
sys 0m23.799s
LiveCD, par_mksquashfs:
real 10m5.572s
user 8m59.921s
sys 0m16.145s
"Home", old mksquashfs:
real 3m42.298s
user 2m49.478s
sys 0m13.675s
"Home", new par_mksquashfs:
real 3m9.178s
user 2m50.699s
sys 0m9.069s
I'll be interested in any performance results obtained, especially from SMP
machines larger than my dual-core AMD box, as this will give an indication of
the scalability of the code. Obviously, I'm also interested in any problems,
deadlocks, low performance etc.
2. UNSQUASHFS
-------------
Unsquashfs now allows you to specify the filename or directory that is to be
extracted from the Squashfs filesystem, rather than always extracting the
entire filesystem. It also has a new "-force" option, and all options can be
specified in a short form (-i rather than -info).
The Unsquashfs usage info is now:
SYNTAX: ./unsquashfs [options] filesystem [directory or file to extract]
-v[ersion] print version, licence and copyright information
-i[nfo] print files as they are unsquashed
-l[s] list filesystem only
-d[est] <pathname> unsquash to <pathname>, default "squashfs-root"
-f[orce] if file already exists then overwrite
To extract a subset of the filesystem, the filename or directory
tree that is to be extracted can now be specified on the command line. The
file/directory should be specified using the full path to the file/directory
as it appears within the Squashfs filesystem. The file/directory will also be
extracted to that position within the specified destination directory.
The new "-force" option forces Unsquashfs to output to the destination
directory even if files or directories already exist. This allows you
to update an existing directory tree, or to Unsquashfs to a partially
filled directory. Without the "-force" option, Unsquashfs will
refuse to overwrite any existing files, or to create any directories if they
already exist. This is done to protect data in case of mistakes, and
so the "-force" option should be used with caution.
SQUASHFS 3.2 - A squashed read-only filesystem for Linux
Copyright 2002-2007 Phillip Lougher <phillip@lougher.org.uk>
Released under the GPL licence (version 2 or later).
Welcome to Squashfs version 3.2. Squashfs 3.2 has support for NFS exporting,
some improvements to the Squashfs tools (Mksquashfs and Unsquashfs), some
major bug fixes, new kernel patches, and various other smaller improvements
and bug fixes. Please see the CHANGES file for a detailed list.
1. MKSQUASHFS
-------------
New command line options:
-no-exports
Squashfs now supports NFS exports. By default the additional
information necessary is added to the filesystem by Mksquashfs. If you
do not wish this extra information, then this option can be specified.
This will save a couple of bytes per file, and the filesystem
will be identical to Squashfs 3.1.
-no-progress
Mksquashfs by default now displays a progress bar. This option disables
it.
2. UNSQUASHFS
-------------
Unsquashfs now supports Squashfs 2.x filesystems.
SQUASHFS 3.3 - A squashed read-only filesystem for Linux
Copyright 2002-2007 Phillip Lougher <phillip@lougher.demon.co.uk>
Released under the GPL licence (version 2 or later).
Welcome to another release of Squashfs. This is the 22nd release in just
over five years of work. Squashfs 3.3 has lots of nice improvements,
both to the filesystem itself (bigger blocks, and sparse files), but
also to the Squashfs-tools Mksquashfs and Unsquashfs. As usual the
CHANGES file has a detailed list of all the improvements.
Following is a description of the changes to the Squashfs tools, usage
guides to the new options, and a summary of the new options.
1. MKSQUASHFS - EXTENDED EXCLUDE FILE HANDLING
----------------------------------------------
1. Extended wildcard pattern matching now supported in exclude files
Enabled by specifying -wildcards option
Supports both anchored and non-anchored exclude files.
1.1 Anchored excludes
Similar to existing exclude files except with wildcards. Exclude
file matches from root of source directories.
Examples:
1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz'
Exclude all files matching "*.gz" in the top level directory "test".
2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*'
Exclude all files beginning with "example" inside directories called
"Test" or "test", that occur inside any top level directory.
Using extended wildcards, negative matching is also possible.
3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz'
Exclude all files matching "*.gz" in top level directory "test",
except those with "data" in the name.
1.2 Non-anchored excludes
By default excludes match from the top level directory, but it is
often useful to exclude a file matching anywhere in the source directories.
For this non-anchored excludes can be used, specified by pre-fixing the
exclude with "...".
Examples:
1. mksquashfs example image.sqsh -wildcards -e '... *.gz'
Exclude files matching "*.gz" anywhere in the source directories.
For example this will match "example.gz", "test/example.gz", and
"test/test/example.gz".
2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz'
Exclude files matching "*.gz" inside directories called "Test" or
"test" that occur anywhere in the source directories.
Again, using extended wildcards, negative matching is also possible.
3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz'
Exclude all files matching "*.gz" anywhere in the source directories,
except those with "data" in the name.
2. Regular expression pattern matching now supported in exclude files
Enabled by specifying -regex option. Identical behaviour to wild
card pattern matching, except patterns are considered to be regular
expressions.
Supports both anchored and non-anchored exclude files.
2. MKSQUASHFS - NEW RECOVERY FILE FEATURE
-----------------------------------------
Recovery files are now created when appending to existing Squashfs
filesystems. This allows the original filesystem to be recovered
if Mksquashfs aborts unexpectedly (i.e. power failure).
The recovery files are called squashfs_recovery_xxx_yyy, where
"xxx" is the name of the filesystem being appended to, and "yyy" is a
number to guarantee filename uniqueness (the PID of the parent Mksquashfs
process).
Normally if Mksquashfs exits correctly the recovery file is deleted to
avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover"
option can be used to recover the filesystem, giving the previously
created recovery file as a parameter, i.e.
mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234
The writing of the recovery file can be disabled by specifying the
"-no-recovery" option.
3. UNSQUASHFS - EXTENDED EXTRACT FILE HANDLING
----------------------------------------------
1. Multiple extract files can now be specified on the command line, and the
files/directories to be extracted can now also be given in a file.
To specify a file containing the extract files use the "-e[f]" option.
2. Extended wildcard pattern matching now supported in extract files
Enabled by default. Similar to existing extract files except with
wildcards.
Examples:
1. unsquashfs image.sqsh 'test/*.gz'
Extract all files matching "*.gz" in the top level directory "test".
2. unsquashfs image.sqsh '[Tt]est/example*'
Extract all files beginning with "example" inside top level directories
called "Test" or "test".
Using extended wildcards, negative matching is also possible.
3. unsquashfs image.sqsh 'test/!(*data*).gz'
Extract all files matching "*.gz" in top level directory "test",
except those with "data" in the name.
3. Regular expression pattern matching now supported in extract files
Enabled by specifying -r[egex] option. Identical behaviour to wild
card pattern matching, except patterns are considered to be regular
expressions.
4. UNSQUASHFS - EXTENDED FILENAME PRINTING
------------------------------------------
Filename printing has been enhanced and Unquashfs can now display filenames
with file attributes ('ls -l' style output).
New options:
-ll[s]
list filesystem with file attributes, but don't unsquash
-li[nfo]
print files as they are unsquashed with file attributes
5. UNSQUASHFS - MISCELLANEOUS OPTIONS
-------------------------------------
-s[tat]
Display the filesystem superblock information. This is useful to
discover the filesystem version, byte ordering, whether it has an
NFS export table, and what options were used to compress
the filesystem.
SQUASHFS 4.0 - A squashed read-only filesystem for Linux
Copyright 2002-2009 Phillip Lougher <phillip@lougher.demon.co.uk>
Released under the GPL licence (version 2 or later).
Welcome to Squashfs 4.0. This is an initial tools only release to
support users of the 2.6.29 kernel, following the mainlining of Squashfs
earlier this year.
Later releases will probably contain kernel patches supporting 4.0
layouts for earlier kernels.
New Mksquashfs options
----------------------
Mksquashfs now supports pseudo files, these allow fake directories, character
and block devices to be specified and added to the Squashfs filesystem being
built, rather than requiring them to be present in the source directories.
This, for example, allows device nodes to be added to the filesystem without
requiring root access.
Two options are supported, -p allows one pseudo file to be specified on the
command line, and -pf allows a pseudo file to be specified containing a
list of pseduo definitions, one per line.
Pseudo device nodes are specified using 7 arguments
Filename type mode uid gid major minor
Where type is either
b - for block devices, and
c - for character devices
mode is the octal mode specifier, similar to that expected by chmod.
Uid and gid can be either specified as a decimal number, or by name.
For example:
/dev/chr_dev c 666 root root 100 1
/dev/blk_dev b 444 0 0 200 200
Directories are specified using 5 arguments
Filename type mode uid gid
Where type is d.
SQUASHFS 4.1 - A squashed read-only filesystem for Linux
Copyright 2002-2010 Phillip Lougher <phillip@lougher.demon.co.uk>
Released under the GPL licence (version 2 or later).
Welcome to Squashfs 4.1. This is a tools only release, support for Squashfs
file systems is in mainline (2.6.29 and later).
New features in Squashfs-tools 4.1
----------------------------------
1. Support for extended attributes
2. Support for LZMA and LZO compression
3. New pseudo file features
Compatiblity
------------
Mksquashfs 4.1 generates 4.0 filesystems. These filesystems are fully
compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are
mountable on 2.6.29 and later kernels.
Extended attributes (xattrs)
----------------------------
Squashfs file systems now have extended attribute support. The
extended attribute implementation has the following features:
1. Layout can store up to 2^48 bytes of compressed xattr data.
2. Number of xattrs per inode unlimited.
3. Total size of xattr data per inode 2^48 bytes of compressed data.
4. Up to 4 Gbytes of data per xattr value.
5. Inline and out-of-line xattr values supported for higher performance
in xattr scanning (listxattr & getxattr), and to allow xattr value
de-duplication.
6. Both whole inode xattr duplicate detection and individual xattr value
duplicate detection supported. These can obviously nest, file C's
xattrs can be a complete duplicate of file B, and file B's xattrs
can be a partial duplicate of file A.
7. Xattr name prefix types stored, allowing the redundant "user.", "trusted."
etc. characters to be eliminated and more concisely stored.
8. Support for files, directories, symbolic links, device nodes, fifos
and sockets.
Extended attribute support is in 2.6.35 and later kernels. File systems
with extended attributes can be mounted on 2.6.29 and later kernels, the
extended attributes will be ignored with a warning.
LZMA and LZO compression
------------------------
Squashfs now supports LZMA and LZO compression.
LZO support is in 2.6.36 and newer kernels. LZMA is not yet in mainline.
New Mksquashfs options
----------------------
-comp <comp>
Select <comp> compression.
The compression algorithms supported by the build of Mksquashfs can be
found by typing mksquashfs without any arguments. The compressors available
are displayed at the end of the help message, e.g.
Compressors available:
gzip (default)
lzma
lzo
The default compression used when -comp isn't specified on the command line
is indicated by "(default)".
-no-xattrs
Don't store extended attributes
-xattrs
Store extended attributes
The default behaviour of Mksquashfs with respect to extended attribute
storage is build time selectable. The Mksquashfs help message indicates
whether extended attributes are stored or not, e.g.
-no-xattrs don't store extended attributes
-xattrs store extended attributes (default)
shows that extended attributes are stored by default, and can be disabled
by the -no-xattrs option.
-no-xattrs don't store extended attributes (default)
-xattrs store extended attributes
shows that extended attributes are not stored by default, storage can be
enabled by the -xattrs option.
-noX
-noXattrCompression
Don't compress extended attributes
New Unsquashfs options
----------------------
-n[o-xattrs]
Don't extract xattrs in filesystem
-x[attrs]
Extract xattrs in filesystem
The default behaviour of Unsquashfs with respect to extended attributes
is build time selectable. The Unsquashfs help message indicates whether
extended attributes are stored or not, e.g.
-no[-xattrs] don't extract xattrs in file system
-x[attrs] extract xattrs in file system (default)
shows that xattrs are extracted by default.
-no[-xattrs] don't extract xattrs in file system (default)
-x[attrs] extract xattrs in file system
shows that xattrs are not extracted by default.
New pseudo file support
-----------------------
Mksquashfs supports pseudo files, these allow fake files, directories, character
and block devices to be specified and added to the Squashfs filesystem being
built, rather than requiring them to be present in the source directories.
This, for example, allows device nodes to be added to the filesystem without
requiring root access.
Mksquashfs 4.1 adds support for "dynamic pseudo files" and a modify operation.
Dynamic pseudo files allow files to be dynamically created when Mksquashfs
is run, their contents being the result of running a command or piece of
shell script. The modifiy operation allows the mode/uid/gid of an existing
file in the source filesystem to be modified.
Two Mksquashfs options are supported, -p allows one pseudo file to be specified
on the command line, and -pf allows a pseudo file to be specified containing a
list of pseduo definitions, one per line.
Pseudo operations
-----------------
1. Creating a dynamic file
--------------------------
Pseudo definition
Filename f mode uid gid command
mode is the octal mode specifier, similar to that expected by chmod.
uid and gid can be either specified as a decimal number, or by name.
command can be an executable or a piece of shell script, and it is executed
by running "/bin/sh -c command". The stdout becomes the contents of
"Filename".
Examples:
Running a basic command
-----------------------
/somedir/dmesg f 444 root root dmesg
creates a file "/somedir/dmesg" containing the output from dmesg.
Executing shell script
----------------------
RELEASE f 444 root root \
if [ ! -e /tmp/ver ]; then \
echo 0 > /tmp/ver; \
fi; \
ver=`cat /tmp/ver`; \
ver=$((ver +1)); \
echo $ver > /tmp/ver; \
echo -n `cat /tmp/release`; \
echo "-dev #"$ver `date` "Build host" `hostname`
Creates a file RELEASE containing the release name, date, build host, and
an incrementing version number. The incrementing version is a side-effect
of executing the shell script, and ensures every time Mksquashfs is run a
new version number is used without requiring any other shell scripting.
The above example also shows that commands can be split across multiple lines
using "\". Obviously as the script will be presented to the shell as a single
line, a semicolon is need to separate individual shell commands within the
shell script.
Reading from a device (or fifo/named socket)
--------------------------------------------
input f 444 root root dd if=/dev/sda1 bs=1024 count=10
Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs
given a device, fifo, or named socket will place that special file within the
Squashfs filesystem, the above allows input from these special files to be
captured and placed in the Squashfs filesystem.
2. Creating a block or character device
---------------------------------------
Pseudo definition
Filename type mode uid gid major minor
Where type is either
b - for block devices, and
c - for character devices
mode is the octal mode specifier, similar to that expected by chmod.
uid and gid can be either specified as a decimal number, or by name.
For example:
/dev/chr_dev c 666 root root 100 1
/dev/blk_dev b 666 0 0 200 200
creates a character device "/dev/chr_dev" with major:minor 100:1 and
a block device "/dev/blk_dev" with major:minor 200:200, both with root
uid/gid and a mode of rw-rw-rw.
3. Creating a directory
-----------------------
Pseudo definition
Filename d mode uid gid
mode is the octal mode specifier, similar to that expected by chmod.
uid and gid can be either specified as a decimal number, or by name.
For example:
/pseudo_dir d 666 root root
creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw.
4. Modifying attributes of an existing file
-------------------------------------------
Pseudo definition
Filename m mode uid gid
mode is the octal mode specifier, similar to that expected by chmod.
uid and gid can be either specified as a decimal number, or by name.
For example:
dmesg m 666 root root
Changes the attributes of the file "dmesg" in the filesystem to have
root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained
from the source filesystem.
SQUASHFS 4.2 - A squashed read-only filesystem for Linux
Copyright 2002-2011 Phillip Lougher <phillip@lougher.demon.co.uk>
Released under the GPL licence (version 2 or later).
Welcome to Squashfs 4.2. This is a tools only release, support for Squashfs
filesystems is in mainline (2.6.29 and later).
New features in Squashfs-tools 4.2
----------------------------------
1. Support for XZ compression
2. Support for compressor specific options
Compatiblity
------------
Mksquashfs 4.2 generates 4.0 filesystems. These filesystems are fully
compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are
mountable on 2.6.29 and later kernels.
XZ compression
--------------
Squashfs now supports XZ compression.
XZ support is in 2.6.38 and newer kernels.
New Mksquashfs options
----------------------
-X<compressor-option>
Compression algorithms can now support compression specific options. These
options are prefixed by -X, and are passed to the compressor for handling.
The compression specific options supported by each compressor can be
found by typing mksquashfs without any arguments. They are displayed at the
end of the help message, e.g.
Compressors available and compressor specific options:
gzip (no options) (default)
lzo (no options)
xz
-Xbcj filter1,filter2,...,filterN
Compress using filter1,filter2,...,filterN in turn
(in addition to no filter), and choose the best compression.
Available filters: x86, arm, armthumb, powerpc, sparc, ia64
-Xdict-size <dict-size>
Use <dict-size> as the XZ dictionary size. The dictionary size
can be specified as a percentage of the block size, or as an
absolute value. The dictionary size must be less than or equal
to the block size and 8192 bytes or larger. It must also be
storable in the xz header as either 2^n or as 2^n+2^(n+1).
Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K
etc.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment