wok annotate linux-libre/stuff/002-squashfs-decompressors-add-boot-time-xz-support.patch @ rev 10166

glibc: remove gawk from glibc (it break the full tazwok build process)
author Christophe Lincoln <pankso@slitaz.org>
date Sat May 21 03:11:49 2011 +0200 (2011-05-21)
parents
children
rev   line source
gokhlayeh@9257 1 From: Lasse Collin <lasse.collin@tukaani.org>
gokhlayeh@9257 2 Date: Thu, 2 Dec 2010 19:14:37 +0000 (+0200)
gokhlayeh@9257 3 Subject: Decompressors: Add boot-time XZ support
gokhlayeh@9257 4 X-Git-Url: http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Fpkl%2Fsquashfs-xz.git;a=commitdiff_plain;h=c64bc9a229b46db75d7761601dd8ca25385a7780
gokhlayeh@9257 5
gokhlayeh@9257 6 Decompressors: Add boot-time XZ support
gokhlayeh@9257 7
gokhlayeh@9257 8 This implements the API defined in <linux/decompress/generic.h>
gokhlayeh@9257 9 which is used for kernel, initramfs, and initrd decompression.
gokhlayeh@9257 10 This patch together with the first patch is enough for
gokhlayeh@9257 11 XZ-compressed initramfs and initrd; XZ-compressed kernel will
gokhlayeh@9257 12 need arch-specific changes.
gokhlayeh@9257 13
gokhlayeh@9257 14 In contrast to other initramfs compression methods, support for
gokhlayeh@9257 15 XZ-compressed initramfs is not enabled by default in usr/Kconfig.
gokhlayeh@9257 16 This is primarily due to the Kconfig options of the xz_dec
gokhlayeh@9257 17 module. It can be good to require that xz_dec is enabled
gokhlayeh@9257 18 separately so the user can select only the BCJ filters he needs
gokhlayeh@9257 19 when EMBEDDED=y.
gokhlayeh@9257 20
gokhlayeh@9257 21 The buffering requirements described in decompress_unxz.c are
gokhlayeh@9257 22 stricter than with gzip, so the relevant changes should be done
gokhlayeh@9257 23 to the arch-specific code when adding support for XZ-compressed
gokhlayeh@9257 24 kernel. Similarly, the heap size in arch-specific pre-boot code
gokhlayeh@9257 25 may need to be increased (30 KiB is enough).
gokhlayeh@9257 26
gokhlayeh@9257 27 The XZ decompressor needs memmove(), memeq() (memcmp() == 0),
gokhlayeh@9257 28 and memzero() (memset(ptr, 0, size)), which aren't available in
gokhlayeh@9257 29 all arch-specific pre-boot environments. I'm including simple
gokhlayeh@9257 30 versions in decompress_unxz.c, but a cleaner solution would
gokhlayeh@9257 31 naturally be nicer.
gokhlayeh@9257 32
gokhlayeh@9257 33 Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
gokhlayeh@9257 34 ---
gokhlayeh@9257 35
gokhlayeh@9257 36 diff --git a/include/linux/decompress/unxz.h b/include/linux/decompress/unxz.h
gokhlayeh@9257 37 new file mode 100644
gokhlayeh@9257 38 index 0000000..41728fc
gokhlayeh@9257 39 --- /dev/null
gokhlayeh@9257 40 +++ b/include/linux/decompress/unxz.h
gokhlayeh@9257 41 @@ -0,0 +1,19 @@
gokhlayeh@9257 42 +/*
gokhlayeh@9257 43 + * Wrapper for decompressing XZ-compressed kernel, initramfs, and initrd
gokhlayeh@9257 44 + *
gokhlayeh@9257 45 + * Author: Lasse Collin <lasse.collin@tukaani.org>
gokhlayeh@9257 46 + *
gokhlayeh@9257 47 + * This file has been put into the public domain.
gokhlayeh@9257 48 + * You can do whatever you want with this file.
gokhlayeh@9257 49 + */
gokhlayeh@9257 50 +
gokhlayeh@9257 51 +#ifndef DECOMPRESS_UNXZ_H
gokhlayeh@9257 52 +#define DECOMPRESS_UNXZ_H
gokhlayeh@9257 53 +
gokhlayeh@9257 54 +int unxz(unsigned char *in, int in_size,
gokhlayeh@9257 55 + int (*fill)(void *dest, unsigned int size),
gokhlayeh@9257 56 + int (*flush)(void *src, unsigned int size),
gokhlayeh@9257 57 + unsigned char *out, int *in_used,
gokhlayeh@9257 58 + void (*error)(char *x));
gokhlayeh@9257 59 +
gokhlayeh@9257 60 +#endif
gokhlayeh@9257 61 diff --git a/init/Kconfig b/init/Kconfig
gokhlayeh@9257 62 index 2de5b1c..d9fbb0f 100644
gokhlayeh@9257 63 --- a/init/Kconfig
gokhlayeh@9257 64 +++ b/init/Kconfig
gokhlayeh@9257 65 @@ -123,13 +123,16 @@ config HAVE_KERNEL_BZIP2
gokhlayeh@9257 66 config HAVE_KERNEL_LZMA
gokhlayeh@9257 67 bool
gokhlayeh@9257 68
gokhlayeh@9257 69 +config HAVE_KERNEL_XZ
gokhlayeh@9257 70 + bool
gokhlayeh@9257 71 +
gokhlayeh@9257 72 config HAVE_KERNEL_LZO
gokhlayeh@9257 73 bool
gokhlayeh@9257 74
gokhlayeh@9257 75 choice
gokhlayeh@9257 76 prompt "Kernel compression mode"
gokhlayeh@9257 77 default KERNEL_GZIP
gokhlayeh@9257 78 - depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_LZO
gokhlayeh@9257 79 + depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO
gokhlayeh@9257 80 help
gokhlayeh@9257 81 The linux kernel is a kind of self-extracting executable.
gokhlayeh@9257 82 Several compression algorithms are available, which differ
gokhlayeh@9257 83 @@ -174,6 +177,21 @@ config KERNEL_LZMA
gokhlayeh@9257 84 two. Compression is slowest. The kernel size is about 33%
gokhlayeh@9257 85 smaller with LZMA in comparison to gzip.
gokhlayeh@9257 86
gokhlayeh@9257 87 +config KERNEL_XZ
gokhlayeh@9257 88 + bool "XZ"
gokhlayeh@9257 89 + depends on HAVE_KERNEL_XZ
gokhlayeh@9257 90 + help
gokhlayeh@9257 91 + XZ uses the LZMA2 algorithm and instruction set specific
gokhlayeh@9257 92 + BCJ filters which can improve compression ratio of executable
gokhlayeh@9257 93 + code. The size of the kernel is about 30% smaller with XZ in
gokhlayeh@9257 94 + comparison to gzip. On architectures for which there is a BCJ
gokhlayeh@9257 95 + filter (i386, x86_64, ARM, IA-64, PowerPC, and SPARC), XZ
gokhlayeh@9257 96 + will create a few percent smaller kernel than plain LZMA.
gokhlayeh@9257 97 +
gokhlayeh@9257 98 + The speed is about the same as with LZMA: The decompression
gokhlayeh@9257 99 + speed of XZ is better than that of bzip2 but worse than gzip
gokhlayeh@9257 100 + and LZO. Compression is slow.
gokhlayeh@9257 101 +
gokhlayeh@9257 102 config KERNEL_LZO
gokhlayeh@9257 103 bool "LZO"
gokhlayeh@9257 104 depends on HAVE_KERNEL_LZO
gokhlayeh@9257 105 diff --git a/lib/Kconfig b/lib/Kconfig
gokhlayeh@9257 106 index 6090314..45960e1 100644
gokhlayeh@9257 107 --- a/lib/Kconfig
gokhlayeh@9257 108 +++ b/lib/Kconfig
gokhlayeh@9257 109 @@ -122,6 +122,9 @@ config DECOMPRESS_BZIP2
gokhlayeh@9257 110 config DECOMPRESS_LZMA
gokhlayeh@9257 111 tristate
gokhlayeh@9257 112
gokhlayeh@9257 113 +config DECOMPRESS_XZ
gokhlayeh@9257 114 + tristate
gokhlayeh@9257 115 +
gokhlayeh@9257 116 config DECOMPRESS_LZO
gokhlayeh@9257 117 select LZO_DECOMPRESS
gokhlayeh@9257 118 tristate
gokhlayeh@9257 119 diff --git a/lib/Makefile b/lib/Makefile
gokhlayeh@9257 120 index f2f98dd..06e3d8a 100644
gokhlayeh@9257 121 --- a/lib/Makefile
gokhlayeh@9257 122 +++ b/lib/Makefile
gokhlayeh@9257 123 @@ -75,6 +75,7 @@ obj-$(CONFIG_RAID6_PQ) += raid6/
gokhlayeh@9257 124 lib-$(CONFIG_DECOMPRESS_GZIP) += decompress_inflate.o
gokhlayeh@9257 125 lib-$(CONFIG_DECOMPRESS_BZIP2) += decompress_bunzip2.o
gokhlayeh@9257 126 lib-$(CONFIG_DECOMPRESS_LZMA) += decompress_unlzma.o
gokhlayeh@9257 127 +lib-$(CONFIG_DECOMPRESS_XZ) += decompress_unxz.o
gokhlayeh@9257 128 lib-$(CONFIG_DECOMPRESS_LZO) += decompress_unlzo.o
gokhlayeh@9257 129
gokhlayeh@9257 130 obj-$(CONFIG_TEXTSEARCH) += textsearch.o
gokhlayeh@9257 131 diff --git a/lib/decompress.c b/lib/decompress.c
gokhlayeh@9257 132 index a760681..3d766b7 100644
gokhlayeh@9257 133 --- a/lib/decompress.c
gokhlayeh@9257 134 +++ b/lib/decompress.c
gokhlayeh@9257 135 @@ -8,6 +8,7 @@
gokhlayeh@9257 136
gokhlayeh@9257 137 #include <linux/decompress/bunzip2.h>
gokhlayeh@9257 138 #include <linux/decompress/unlzma.h>
gokhlayeh@9257 139 +#include <linux/decompress/unxz.h>
gokhlayeh@9257 140 #include <linux/decompress/inflate.h>
gokhlayeh@9257 141 #include <linux/decompress/unlzo.h>
gokhlayeh@9257 142
gokhlayeh@9257 143 @@ -23,6 +24,9 @@
gokhlayeh@9257 144 #ifndef CONFIG_DECOMPRESS_LZMA
gokhlayeh@9257 145 # define unlzma NULL
gokhlayeh@9257 146 #endif
gokhlayeh@9257 147 +#ifndef CONFIG_DECOMPRESS_XZ
gokhlayeh@9257 148 +# define unxz NULL
gokhlayeh@9257 149 +#endif
gokhlayeh@9257 150 #ifndef CONFIG_DECOMPRESS_LZO
gokhlayeh@9257 151 # define unlzo NULL
gokhlayeh@9257 152 #endif
gokhlayeh@9257 153 @@ -36,6 +40,7 @@ static const struct compress_format {
gokhlayeh@9257 154 { {037, 0236}, "gzip", gunzip },
gokhlayeh@9257 155 { {0x42, 0x5a}, "bzip2", bunzip2 },
gokhlayeh@9257 156 { {0x5d, 0x00}, "lzma", unlzma },
gokhlayeh@9257 157 + { {0xfd, 0x37}, "xz", unxz },
gokhlayeh@9257 158 { {0x89, 0x4c}, "lzo", unlzo },
gokhlayeh@9257 159 { {0, 0}, NULL, NULL }
gokhlayeh@9257 160 };
gokhlayeh@9257 161 diff --git a/lib/decompress_unxz.c b/lib/decompress_unxz.c
gokhlayeh@9257 162 new file mode 100644
gokhlayeh@9257 163 index 0000000..cecd23d
gokhlayeh@9257 164 --- /dev/null
gokhlayeh@9257 165 +++ b/lib/decompress_unxz.c
gokhlayeh@9257 166 @@ -0,0 +1,397 @@
gokhlayeh@9257 167 +/*
gokhlayeh@9257 168 + * Wrapper for decompressing XZ-compressed kernel, initramfs, and initrd
gokhlayeh@9257 169 + *
gokhlayeh@9257 170 + * Author: Lasse Collin <lasse.collin@tukaani.org>
gokhlayeh@9257 171 + *
gokhlayeh@9257 172 + * This file has been put into the public domain.
gokhlayeh@9257 173 + * You can do whatever you want with this file.
gokhlayeh@9257 174 + */
gokhlayeh@9257 175 +
gokhlayeh@9257 176 +/*
gokhlayeh@9257 177 + * Important notes about in-place decompression
gokhlayeh@9257 178 + *
gokhlayeh@9257 179 + * At least on x86, the kernel is decompressed in place: the compressed data
gokhlayeh@9257 180 + * is placed to the end of the output buffer, and the decompressor overwrites
gokhlayeh@9257 181 + * most of the compressed data. There must be enough safety margin to
gokhlayeh@9257 182 + * guarantee that the write position is always behind the read position.
gokhlayeh@9257 183 + *
gokhlayeh@9257 184 + * The safety margin for XZ with LZMA2 or BCJ+LZMA2 is calculated below.
gokhlayeh@9257 185 + * Note that the margin with XZ is bigger than with Deflate (gzip)!
gokhlayeh@9257 186 + *
gokhlayeh@9257 187 + * The worst case for in-place decompression is that the beginning of
gokhlayeh@9257 188 + * the file is compressed extremely well, and the rest of the file is
gokhlayeh@9257 189 + * uncompressible. Thus, we must look for worst-case expansion when the
gokhlayeh@9257 190 + * compressor is encoding uncompressible data.
gokhlayeh@9257 191 + *
gokhlayeh@9257 192 + * The structure of the .xz file in case of a compresed kernel is as follows.
gokhlayeh@9257 193 + * Sizes (as bytes) of the fields are in parenthesis.
gokhlayeh@9257 194 + *
gokhlayeh@9257 195 + * Stream Header (12)
gokhlayeh@9257 196 + * Block Header:
gokhlayeh@9257 197 + * Block Header (8-12)
gokhlayeh@9257 198 + * Compressed Data (N)
gokhlayeh@9257 199 + * Block Padding (0-3)
gokhlayeh@9257 200 + * CRC32 (4)
gokhlayeh@9257 201 + * Index (8-20)
gokhlayeh@9257 202 + * Stream Footer (12)
gokhlayeh@9257 203 + *
gokhlayeh@9257 204 + * Normally there is exactly one Block, but let's assume that there are
gokhlayeh@9257 205 + * 2-4 Blocks just in case. Because Stream Header and also Block Header
gokhlayeh@9257 206 + * of the first Block don't make the decompressor produce any uncompressed
gokhlayeh@9257 207 + * data, we can ignore them from our calculations. Block Headers of possible
gokhlayeh@9257 208 + * additional Blocks have to be taken into account still. With these
gokhlayeh@9257 209 + * assumptions, it is safe to assume that the total header overhead is
gokhlayeh@9257 210 + * less than 128 bytes.
gokhlayeh@9257 211 + *
gokhlayeh@9257 212 + * Compressed Data contains LZMA2 or BCJ+LZMA2 encoded data. Since BCJ
gokhlayeh@9257 213 + * doesn't change the size of the data, it is enough to calculate the
gokhlayeh@9257 214 + * safety margin for LZMA2.
gokhlayeh@9257 215 + *
gokhlayeh@9257 216 + * LZMA2 stores the data in chunks. Each chunk has a header whose size is
gokhlayeh@9257 217 + * a maximum of 6 bytes, but to get round 2^n numbers, let's assume that
gokhlayeh@9257 218 + * the maximum chunk header size is 8 bytes. After the chunk header, there
gokhlayeh@9257 219 + * may be up to 64 KiB of actual payload in the chunk. Often the payload is
gokhlayeh@9257 220 + * quite a bit smaller though; to be safe, let's assume that an average
gokhlayeh@9257 221 + * chunk has only 32 KiB of payload.
gokhlayeh@9257 222 + *
gokhlayeh@9257 223 + * The maximum uncompressed size of the payload is 2 MiB. The minimum
gokhlayeh@9257 224 + * uncompressed size of the payload is in practice never less than the
gokhlayeh@9257 225 + * payload size itself. The LZMA2 format would allow uncompressed size
gokhlayeh@9257 226 + * to be less than the payload size, but no sane compressor creates such
gokhlayeh@9257 227 + * files. LZMA2 supports storing uncompressible data in uncompressed form,
gokhlayeh@9257 228 + * so there's never a need to create payloads whose uncompressed size is
gokhlayeh@9257 229 + * smaller than the compressed size.
gokhlayeh@9257 230 + *
gokhlayeh@9257 231 + * The assumption, that the uncompressed size of the payload is never
gokhlayeh@9257 232 + * smaller than the payload itself, is valid only when talking about
gokhlayeh@9257 233 + * the payload as a whole. It is possible that the payload has parts where
gokhlayeh@9257 234 + * the decompressor consumes more input than it produces output. Calculating
gokhlayeh@9257 235 + * the worst case for this would be tricky. Instead of trying to do that,
gokhlayeh@9257 236 + * let's simply make sure that the decompressor never overwrites any bytes
gokhlayeh@9257 237 + * of the payload which it is currently reading.
gokhlayeh@9257 238 + *
gokhlayeh@9257 239 + * Now we have enough information to calculate the safety margin. We need
gokhlayeh@9257 240 + * - 128 bytes for the .xz file format headers;
gokhlayeh@9257 241 + * - 8 bytes per every 32 KiB of uncompressed size (one LZMA2 chunk header
gokhlayeh@9257 242 + * per chunk, each chunk having average payload size of 32 KiB); and
gokhlayeh@9257 243 + * - 64 KiB (biggest possible LZMA2 chunk payload size) to make sure that
gokhlayeh@9257 244 + * the decompressor never overwrites anything from the LZMA2 chunk
gokhlayeh@9257 245 + * payload it is currently reading.
gokhlayeh@9257 246 + *
gokhlayeh@9257 247 + * We get the following formula:
gokhlayeh@9257 248 + *
gokhlayeh@9257 249 + * safety_margin = 128 + uncompressed_size * 8 / 32768 + 65536
gokhlayeh@9257 250 + * = 128 + (uncompressed_size >> 12) + 65536
gokhlayeh@9257 251 + *
gokhlayeh@9257 252 + * For comparision, according to arch/x86/boot/compressed/misc.c, the
gokhlayeh@9257 253 + * equivalent formula for Deflate is this:
gokhlayeh@9257 254 + *
gokhlayeh@9257 255 + * safety_margin = 18 + (uncompressed_size >> 12) + 32768
gokhlayeh@9257 256 + *
gokhlayeh@9257 257 + * Thus, when updating Deflate-only in-place kernel decompressor to
gokhlayeh@9257 258 + * support XZ, the fixed overhead has to be increased from 18+32768 bytes
gokhlayeh@9257 259 + * to 128+65536 bytes.
gokhlayeh@9257 260 + */
gokhlayeh@9257 261 +
gokhlayeh@9257 262 +/*
gokhlayeh@9257 263 + * STATIC is defined to "static" if we are being built for kernel
gokhlayeh@9257 264 + * decompression (pre-boot code). <linux/decompress/mm.h> will define
gokhlayeh@9257 265 + * STATIC to empty if it wasn't already defined. Since we will need to
gokhlayeh@9257 266 + * know later if we are being used for kernel decompression, we define
gokhlayeh@9257 267 + * XZ_PREBOOT here.
gokhlayeh@9257 268 + */
gokhlayeh@9257 269 +#ifdef STATIC
gokhlayeh@9257 270 +# define XZ_PREBOOT
gokhlayeh@9257 271 +#endif
gokhlayeh@9257 272 +#ifdef __KERNEL__
gokhlayeh@9257 273 +# include <linux/decompress/mm.h>
gokhlayeh@9257 274 +#endif
gokhlayeh@9257 275 +#define XZ_EXTERN STATIC
gokhlayeh@9257 276 +
gokhlayeh@9257 277 +#ifndef XZ_PREBOOT
gokhlayeh@9257 278 +# include <linux/slab.h>
gokhlayeh@9257 279 +# include <linux/xz.h>
gokhlayeh@9257 280 +#else
gokhlayeh@9257 281 +/*
gokhlayeh@9257 282 + * Use the internal CRC32 code instead of kernel's CRC32 module, which
gokhlayeh@9257 283 + * is not available in early phase of booting.
gokhlayeh@9257 284 + */
gokhlayeh@9257 285 +#define XZ_INTERNAL_CRC32 1
gokhlayeh@9257 286 +
gokhlayeh@9257 287 +/*
gokhlayeh@9257 288 + * For boot time use, we enable only the BCJ filter of the current
gokhlayeh@9257 289 + * architecture or none if no BCJ filter is available for the architecture.
gokhlayeh@9257 290 + */
gokhlayeh@9257 291 +#ifdef CONFIG_X86
gokhlayeh@9257 292 +# define XZ_DEC_X86
gokhlayeh@9257 293 +#endif
gokhlayeh@9257 294 +#ifdef CONFIG_PPC
gokhlayeh@9257 295 +# define XZ_DEC_POWERPC
gokhlayeh@9257 296 +#endif
gokhlayeh@9257 297 +#ifdef CONFIG_ARM
gokhlayeh@9257 298 +# define XZ_DEC_ARM
gokhlayeh@9257 299 +#endif
gokhlayeh@9257 300 +#ifdef CONFIG_IA64
gokhlayeh@9257 301 +# define XZ_DEC_IA64
gokhlayeh@9257 302 +#endif
gokhlayeh@9257 303 +#ifdef CONFIG_SPARC
gokhlayeh@9257 304 +# define XZ_DEC_SPARC
gokhlayeh@9257 305 +#endif
gokhlayeh@9257 306 +
gokhlayeh@9257 307 +/*
gokhlayeh@9257 308 + * This will get the basic headers so that memeq() and others
gokhlayeh@9257 309 + * can be defined.
gokhlayeh@9257 310 + */
gokhlayeh@9257 311 +#include "xz/xz_private.h"
gokhlayeh@9257 312 +
gokhlayeh@9257 313 +/*
gokhlayeh@9257 314 + * Replace the normal allocation functions with the versions from
gokhlayeh@9257 315 + * <linux/decompress/mm.h>. vfree() needs to support vfree(NULL)
gokhlayeh@9257 316 + * when XZ_DYNALLOC is used, but the pre-boot free() doesn't support it.
gokhlayeh@9257 317 + * Workaround it here because the other decompressors don't need it.
gokhlayeh@9257 318 + */
gokhlayeh@9257 319 +#undef kmalloc
gokhlayeh@9257 320 +#undef kfree
gokhlayeh@9257 321 +#undef vmalloc
gokhlayeh@9257 322 +#undef vfree
gokhlayeh@9257 323 +#define kmalloc(size, flags) malloc(size)
gokhlayeh@9257 324 +#define kfree(ptr) free(ptr)
gokhlayeh@9257 325 +#define vmalloc(size) malloc(size)
gokhlayeh@9257 326 +#define vfree(ptr) do { if (ptr != NULL) free(ptr); } while (0)
gokhlayeh@9257 327 +
gokhlayeh@9257 328 +/*
gokhlayeh@9257 329 + * FIXME: Not all basic memory functions are provided in architecture-specific
gokhlayeh@9257 330 + * files (yet). We define our own versions here for now, but this should be
gokhlayeh@9257 331 + * only a temporary solution.
gokhlayeh@9257 332 + *
gokhlayeh@9257 333 + * memeq and memzero are not used much and any remotely sane implementation
gokhlayeh@9257 334 + * is fast enough. memcpy/memmove speed matters in multi-call mode, but
gokhlayeh@9257 335 + * the kernel image is decompressed in single-call mode, in which only
gokhlayeh@9257 336 + * memcpy speed can matter and only if there is a lot of uncompressible data
gokhlayeh@9257 337 + * (LZMA2 stores uncompressible chunks in uncompressed form). Thus, the
gokhlayeh@9257 338 + * functions below should just be kept small; it's probably not worth
gokhlayeh@9257 339 + * optimizing for speed.
gokhlayeh@9257 340 + */
gokhlayeh@9257 341 +
gokhlayeh@9257 342 +#ifndef memeq
gokhlayeh@9257 343 +static bool memeq(const void *a, const void *b, size_t size)
gokhlayeh@9257 344 +{
gokhlayeh@9257 345 + const uint8_t *x = a;
gokhlayeh@9257 346 + const uint8_t *y = b;
gokhlayeh@9257 347 + size_t i;
gokhlayeh@9257 348 +
gokhlayeh@9257 349 + for (i = 0; i < size; ++i)
gokhlayeh@9257 350 + if (x[i] != y[i])
gokhlayeh@9257 351 + return false;
gokhlayeh@9257 352 +
gokhlayeh@9257 353 + return true;
gokhlayeh@9257 354 +}
gokhlayeh@9257 355 +#endif
gokhlayeh@9257 356 +
gokhlayeh@9257 357 +#ifndef memzero
gokhlayeh@9257 358 +static void memzero(void *buf, size_t size)
gokhlayeh@9257 359 +{
gokhlayeh@9257 360 + uint8_t *b = buf;
gokhlayeh@9257 361 + uint8_t *e = b + size;
gokhlayeh@9257 362 +
gokhlayeh@9257 363 + while (b != e)
gokhlayeh@9257 364 + *b++ = '\0';
gokhlayeh@9257 365 +}
gokhlayeh@9257 366 +#endif
gokhlayeh@9257 367 +
gokhlayeh@9257 368 +#ifndef memmove
gokhlayeh@9257 369 +/* Not static to avoid a conflict with the prototype in the Linux headers. */
gokhlayeh@9257 370 +void *memmove(void *dest, const void *src, size_t size)
gokhlayeh@9257 371 +{
gokhlayeh@9257 372 + uint8_t *d = dest;
gokhlayeh@9257 373 + const uint8_t *s = src;
gokhlayeh@9257 374 + size_t i;
gokhlayeh@9257 375 +
gokhlayeh@9257 376 + if (d < s) {
gokhlayeh@9257 377 + for (i = 0; i < size; ++i)
gokhlayeh@9257 378 + d[i] = s[i];
gokhlayeh@9257 379 + } else if (d > s) {
gokhlayeh@9257 380 + i = size;
gokhlayeh@9257 381 + while (i-- > 0)
gokhlayeh@9257 382 + d[i] = s[i];
gokhlayeh@9257 383 + }
gokhlayeh@9257 384 +
gokhlayeh@9257 385 + return dest;
gokhlayeh@9257 386 +}
gokhlayeh@9257 387 +#endif
gokhlayeh@9257 388 +
gokhlayeh@9257 389 +/*
gokhlayeh@9257 390 + * Since we need memmove anyway, would use it as memcpy too.
gokhlayeh@9257 391 + * Commented out for now to avoid breaking things.
gokhlayeh@9257 392 + */
gokhlayeh@9257 393 +/*
gokhlayeh@9257 394 +#ifndef memcpy
gokhlayeh@9257 395 +# define memcpy memmove
gokhlayeh@9257 396 +#endif
gokhlayeh@9257 397 +*/
gokhlayeh@9257 398 +
gokhlayeh@9257 399 +#include "xz/xz_crc32.c"
gokhlayeh@9257 400 +#include "xz/xz_dec_stream.c"
gokhlayeh@9257 401 +#include "xz/xz_dec_lzma2.c"
gokhlayeh@9257 402 +#include "xz/xz_dec_bcj.c"
gokhlayeh@9257 403 +
gokhlayeh@9257 404 +#endif /* XZ_PREBOOT */
gokhlayeh@9257 405 +
gokhlayeh@9257 406 +/* Size of the input and output buffers in multi-call mode */
gokhlayeh@9257 407 +#define XZ_IOBUF_SIZE 4096
gokhlayeh@9257 408 +
gokhlayeh@9257 409 +/*
gokhlayeh@9257 410 + * This function implements the API defined in <linux/decompress/generic.h>.
gokhlayeh@9257 411 + *
gokhlayeh@9257 412 + * This wrapper will automatically choose single-call or multi-call mode
gokhlayeh@9257 413 + * of the native XZ decoder API. The single-call mode can be used only when
gokhlayeh@9257 414 + * both input and output buffers are available as a single chunk, i.e. when
gokhlayeh@9257 415 + * fill() and flush() won't be used.
gokhlayeh@9257 416 + */
gokhlayeh@9257 417 +STATIC int INIT unxz(unsigned char *in, int in_size,
gokhlayeh@9257 418 + int (*fill)(void *dest, unsigned int size),
gokhlayeh@9257 419 + int (*flush)(void *src, unsigned int size),
gokhlayeh@9257 420 + unsigned char *out, int *in_used,
gokhlayeh@9257 421 + void (*error)(char *x))
gokhlayeh@9257 422 +{
gokhlayeh@9257 423 + struct xz_buf b;
gokhlayeh@9257 424 + struct xz_dec *s;
gokhlayeh@9257 425 + enum xz_ret ret;
gokhlayeh@9257 426 + bool must_free_in = false;
gokhlayeh@9257 427 +
gokhlayeh@9257 428 +#if XZ_INTERNAL_CRC32
gokhlayeh@9257 429 + xz_crc32_init();
gokhlayeh@9257 430 +#endif
gokhlayeh@9257 431 +
gokhlayeh@9257 432 + if (in_used != NULL)
gokhlayeh@9257 433 + *in_used = 0;
gokhlayeh@9257 434 +
gokhlayeh@9257 435 + if (fill == NULL && flush == NULL)
gokhlayeh@9257 436 + s = xz_dec_init(XZ_SINGLE, 0);
gokhlayeh@9257 437 + else
gokhlayeh@9257 438 + s = xz_dec_init(XZ_DYNALLOC, (uint32_t)-1);
gokhlayeh@9257 439 +
gokhlayeh@9257 440 + if (s == NULL)
gokhlayeh@9257 441 + goto error_alloc_state;
gokhlayeh@9257 442 +
gokhlayeh@9257 443 + if (flush == NULL) {
gokhlayeh@9257 444 + b.out = out;
gokhlayeh@9257 445 + b.out_size = (size_t)-1;
gokhlayeh@9257 446 + } else {
gokhlayeh@9257 447 + b.out_size = XZ_IOBUF_SIZE;
gokhlayeh@9257 448 + b.out = malloc(XZ_IOBUF_SIZE);
gokhlayeh@9257 449 + if (b.out == NULL)
gokhlayeh@9257 450 + goto error_alloc_out;
gokhlayeh@9257 451 + }
gokhlayeh@9257 452 +
gokhlayeh@9257 453 + if (in == NULL) {
gokhlayeh@9257 454 + must_free_in = true;
gokhlayeh@9257 455 + in = malloc(XZ_IOBUF_SIZE);
gokhlayeh@9257 456 + if (in == NULL)
gokhlayeh@9257 457 + goto error_alloc_in;
gokhlayeh@9257 458 + }
gokhlayeh@9257 459 +
gokhlayeh@9257 460 + b.in = in;
gokhlayeh@9257 461 + b.in_pos = 0;
gokhlayeh@9257 462 + b.in_size = in_size;
gokhlayeh@9257 463 + b.out_pos = 0;
gokhlayeh@9257 464 +
gokhlayeh@9257 465 + if (fill == NULL && flush == NULL) {
gokhlayeh@9257 466 + ret = xz_dec_run(s, &b);
gokhlayeh@9257 467 + } else {
gokhlayeh@9257 468 + do {
gokhlayeh@9257 469 + if (b.in_pos == b.in_size && fill != NULL) {
gokhlayeh@9257 470 + if (in_used != NULL)
gokhlayeh@9257 471 + *in_used += b.in_pos;
gokhlayeh@9257 472 +
gokhlayeh@9257 473 + b.in_pos = 0;
gokhlayeh@9257 474 +
gokhlayeh@9257 475 + in_size = fill(in, XZ_IOBUF_SIZE);
gokhlayeh@9257 476 + if (in_size < 0) {
gokhlayeh@9257 477 + /*
gokhlayeh@9257 478 + * This isn't an optimal error code
gokhlayeh@9257 479 + * but it probably isn't worth making
gokhlayeh@9257 480 + * a new one either.
gokhlayeh@9257 481 + */
gokhlayeh@9257 482 + ret = XZ_BUF_ERROR;
gokhlayeh@9257 483 + break;
gokhlayeh@9257 484 + }
gokhlayeh@9257 485 +
gokhlayeh@9257 486 + b.in_size = in_size;
gokhlayeh@9257 487 + }
gokhlayeh@9257 488 +
gokhlayeh@9257 489 + ret = xz_dec_run(s, &b);
gokhlayeh@9257 490 +
gokhlayeh@9257 491 + if (flush != NULL && (b.out_pos == b.out_size
gokhlayeh@9257 492 + || (ret != XZ_OK && b.out_pos > 0))) {
gokhlayeh@9257 493 + /*
gokhlayeh@9257 494 + * Setting ret here may hide an error
gokhlayeh@9257 495 + * returned by xz_dec_run(), but probably
gokhlayeh@9257 496 + * it's not too bad.
gokhlayeh@9257 497 + */
gokhlayeh@9257 498 + if (flush(b.out, b.out_pos) != (int)b.out_pos)
gokhlayeh@9257 499 + ret = XZ_BUF_ERROR;
gokhlayeh@9257 500 +
gokhlayeh@9257 501 + b.out_pos = 0;
gokhlayeh@9257 502 + }
gokhlayeh@9257 503 + } while (ret == XZ_OK);
gokhlayeh@9257 504 +
gokhlayeh@9257 505 + if (must_free_in)
gokhlayeh@9257 506 + free(in);
gokhlayeh@9257 507 +
gokhlayeh@9257 508 + if (flush != NULL)
gokhlayeh@9257 509 + free(b.out);
gokhlayeh@9257 510 + }
gokhlayeh@9257 511 +
gokhlayeh@9257 512 + if (in_used != NULL)
gokhlayeh@9257 513 + *in_used += b.in_pos;
gokhlayeh@9257 514 +
gokhlayeh@9257 515 + xz_dec_end(s);
gokhlayeh@9257 516 +
gokhlayeh@9257 517 + switch (ret) {
gokhlayeh@9257 518 + case XZ_STREAM_END:
gokhlayeh@9257 519 + return 0;
gokhlayeh@9257 520 +
gokhlayeh@9257 521 + case XZ_MEM_ERROR:
gokhlayeh@9257 522 + /* This can occur only in multi-call mode. */
gokhlayeh@9257 523 + error("XZ decompressor ran out of memory");
gokhlayeh@9257 524 + break;
gokhlayeh@9257 525 +
gokhlayeh@9257 526 + case XZ_FORMAT_ERROR:
gokhlayeh@9257 527 + error("Input is not in the XZ format (wrong magic bytes)");
gokhlayeh@9257 528 + break;
gokhlayeh@9257 529 +
gokhlayeh@9257 530 + case XZ_OPTIONS_ERROR:
gokhlayeh@9257 531 + error("Input was encoded with settings that are not "
gokhlayeh@9257 532 + "supported by this XZ decoder");
gokhlayeh@9257 533 + break;
gokhlayeh@9257 534 +
gokhlayeh@9257 535 + case XZ_DATA_ERROR:
gokhlayeh@9257 536 + case XZ_BUF_ERROR:
gokhlayeh@9257 537 + error("XZ-compressed data is corrupt");
gokhlayeh@9257 538 + break;
gokhlayeh@9257 539 +
gokhlayeh@9257 540 + default:
gokhlayeh@9257 541 + error("Bug in the XZ decompressor");
gokhlayeh@9257 542 + break;
gokhlayeh@9257 543 + }
gokhlayeh@9257 544 +
gokhlayeh@9257 545 + return -1;
gokhlayeh@9257 546 +
gokhlayeh@9257 547 +error_alloc_in:
gokhlayeh@9257 548 + if (flush != NULL)
gokhlayeh@9257 549 + free(b.out);
gokhlayeh@9257 550 +
gokhlayeh@9257 551 +error_alloc_out:
gokhlayeh@9257 552 + xz_dec_end(s);
gokhlayeh@9257 553 +
gokhlayeh@9257 554 +error_alloc_state:
gokhlayeh@9257 555 + error("XZ decompressor ran out of memory");
gokhlayeh@9257 556 + return -1;
gokhlayeh@9257 557 +}
gokhlayeh@9257 558 +
gokhlayeh@9257 559 +/*
gokhlayeh@9257 560 + * This macro is used by architecture-specific files to decompress
gokhlayeh@9257 561 + * the kernel image.
gokhlayeh@9257 562 + */
gokhlayeh@9257 563 +#define decompress unxz
gokhlayeh@9257 564 diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh
gokhlayeh@9257 565 index 5958fff..55caecd 100644
gokhlayeh@9257 566 --- a/scripts/gen_initramfs_list.sh
gokhlayeh@9257 567 +++ b/scripts/gen_initramfs_list.sh
gokhlayeh@9257 568 @@ -243,6 +243,8 @@ case "$arg" in
gokhlayeh@9257 569 echo "$output_file" | grep -q "\.gz$" && compr="gzip -9 -f"
gokhlayeh@9257 570 echo "$output_file" | grep -q "\.bz2$" && compr="bzip2 -9 -f"
gokhlayeh@9257 571 echo "$output_file" | grep -q "\.lzma$" && compr="lzma -9 -f"
gokhlayeh@9257 572 + echo "$output_file" | grep -q "\.xz$" && \
gokhlayeh@9257 573 + compr="xz --check=crc32 --lzma2=dict=1MiB"
gokhlayeh@9257 574 echo "$output_file" | grep -q "\.lzo$" && compr="lzop -9 -f"
gokhlayeh@9257 575 echo "$output_file" | grep -q "\.cpio$" && compr="cat"
gokhlayeh@9257 576 shift
gokhlayeh@9257 577 diff --git a/usr/Kconfig b/usr/Kconfig
gokhlayeh@9257 578 index e2721f5..9f51a29 100644
gokhlayeh@9257 579 --- a/usr/Kconfig
gokhlayeh@9257 580 +++ b/usr/Kconfig
gokhlayeh@9257 581 @@ -72,6 +72,18 @@ config RD_LZMA
gokhlayeh@9257 582 Support loading of a LZMA encoded initial ramdisk or cpio buffer
gokhlayeh@9257 583 If unsure, say N.
gokhlayeh@9257 584
gokhlayeh@9257 585 +config RD_XZ
gokhlayeh@9257 586 + bool "Support initial ramdisks compressed using XZ"
gokhlayeh@9257 587 + depends on BLK_DEV_INITRD && XZ_DEC=y
gokhlayeh@9257 588 + select DECOMPRESS_XZ
gokhlayeh@9257 589 + help
gokhlayeh@9257 590 + Support loading of a XZ encoded initial ramdisk or cpio buffer.
gokhlayeh@9257 591 +
gokhlayeh@9257 592 + If this option is inactive, say Y to "XZ decompression support"
gokhlayeh@9257 593 + under "Library routines" first.
gokhlayeh@9257 594 +
gokhlayeh@9257 595 + If unsure, say N.
gokhlayeh@9257 596 +
gokhlayeh@9257 597 config RD_LZO
gokhlayeh@9257 598 bool "Support initial ramdisks compressed using LZO" if EMBEDDED
gokhlayeh@9257 599 default !EMBEDDED
gokhlayeh@9257 600 @@ -139,6 +151,15 @@ config INITRAMFS_COMPRESSION_LZMA
gokhlayeh@9257 601 three. Compression is slowest. The initramfs size is about 33%
gokhlayeh@9257 602 smaller with LZMA in comparison to gzip.
gokhlayeh@9257 603
gokhlayeh@9257 604 +config INITRAMFS_COMPRESSION_XZ
gokhlayeh@9257 605 + bool "XZ"
gokhlayeh@9257 606 + depends on RD_XZ
gokhlayeh@9257 607 + help
gokhlayeh@9257 608 + XZ uses the LZMA2 algorithm. The initramfs size is about 30%
gokhlayeh@9257 609 + smaller with XZ in comparison to gzip. Decompression speed
gokhlayeh@9257 610 + is better than that of bzip2 but worse than gzip and LZO.
gokhlayeh@9257 611 + Compression is slow.
gokhlayeh@9257 612 +
gokhlayeh@9257 613 config INITRAMFS_COMPRESSION_LZO
gokhlayeh@9257 614 bool "LZO"
gokhlayeh@9257 615 depends on RD_LZO
gokhlayeh@9257 616 diff --git a/usr/Makefile b/usr/Makefile
gokhlayeh@9257 617 index 6b4b6da..5845a13 100644
gokhlayeh@9257 618 --- a/usr/Makefile
gokhlayeh@9257 619 +++ b/usr/Makefile
gokhlayeh@9257 620 @@ -15,6 +15,9 @@ suffix_$(CONFIG_INITRAMFS_COMPRESSION_BZIP2) = .bz2
gokhlayeh@9257 621 # Lzma
gokhlayeh@9257 622 suffix_$(CONFIG_INITRAMFS_COMPRESSION_LZMA) = .lzma
gokhlayeh@9257 623
gokhlayeh@9257 624 +# XZ
gokhlayeh@9257 625 +suffix_$(CONFIG_INITRAMFS_COMPRESSION_XZ) = .xz
gokhlayeh@9257 626 +
gokhlayeh@9257 627 # Lzo
gokhlayeh@9257 628 suffix_$(CONFIG_INITRAMFS_COMPRESSION_LZO) = .lzo
gokhlayeh@9257 629
gokhlayeh@9257 630 @@ -48,7 +51,7 @@ endif
gokhlayeh@9257 631 quiet_cmd_initfs = GEN $@
gokhlayeh@9257 632 cmd_initfs = $(initramfs) -o $@ $(ramfs-args) $(ramfs-input)
gokhlayeh@9257 633
gokhlayeh@9257 634 -targets := initramfs_data.cpio.gz initramfs_data.cpio.bz2 initramfs_data.cpio.lzma initramfs_data.cpio.lzo initramfs_data.cpio
gokhlayeh@9257 635 +targets := initramfs_data.cpio.gz initramfs_data.cpio.bz2 initramfs_data.cpio.lzma initramfs_data.cpio.xz initramfs_data.cpio.lzo initramfs_data.cpio
gokhlayeh@9257 636 # do not try to update files included in initramfs
gokhlayeh@9257 637 $(deps_initramfs): ;
gokhlayeh@9257 638