wok-tiny: 5d8744871068 kernel-modular/stuff/001-squashfs-decompressors-add-xz-decompressor-module.patch

wok-tiny view kernel-modular/stuff/001-squashfs-decompressors-add-xz-decompressor-module.patch @ rev 1

Add kernel

author	Pascal Bellard <pascal.bellard@slitaz.org>
date	Tue Feb 01 09:37:33 2011 +0100 (2011-02-01)
parents
children

line source

1 From: Lasse Collin <lasse.collin@tukaani.org>

2 Date: Thu, 2 Dec 2010 19:14:19 +0000 (+0200)

3 Subject: Decompressors: Add XZ decompressor module

4 X-Git-Url: http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Fpkl%2Fsquashfs-xz.git;a=commitdiff_plain;h=3dbc3fe7878e53b43064a12d4ab31ca4c18ce85f

6 Decompressors: Add XZ decompressor module

8 In userspace, the .lzma format has become mostly a legacy

9 file format that got superseded by the .xz format. Similarly,

10 LZMA Utils was superseded by XZ Utils.

12 These patches add support for XZ decompression into

13 the kernel. Most of the code is as is from XZ Embedded

14 <http://tukaani.org/xz/embedded.html>. It was written for

15 the Linux kernel but is usable in other projects too.

17 Advantages of XZ over the current LZMA code in the kernel:

18 - Nice API that can be used by other kernel modules; it's

19 not limited to kernel, initramfs, and initrd decompression.

20 - Integrity check support (CRC32)

21 - BCJ filters improve compression of executable code on

22 certain architectures. These together with LZMA2 can

23 produce a few percent smaller kernel or Squashfs images

24 than plain LZMA without making the decompression slower.

26 This patch: Add the main decompression code (xz_dec), testing

27 module (xz_dec_test), wrapper script (xz_wrap.sh) for the xz

28 command line tool, and documentation. The xz_dec module is

29 enough to have a usable XZ decompressor e.g. for Squashfs.

31 Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>

32 ---

34 diff --git a/Documentation/xz.txt b/Documentation/xz.txt

35 new file mode 100644

36 index 0000000..68329ac

37 --- /dev/null

38 +++ b/Documentation/xz.txt

39 @@ -0,0 +1,122 @@

40 +

41 +XZ data compression in Linux

42 +============================

43 +

44 +Introduction

45 +

46 + XZ is a general purpose data compression format with high compression

47 + ratio and relatively fast decompression. The primary compression

48 + algorithm (filter) is LZMA2. Additional filters can be used to improve

49 + compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters

50 + improve compression ratio of executable data.

51 +

52 + The XZ decompressor in Linux is called XZ Embedded. It supports

53 + the LZMA2 filter and optionally also BCJ filters. CRC32 is supported

54 + for integrity checking. The home page of XZ Embedded is at

55 + <http://tukaani.org/xz/embedded.html>, where you can find the

56 + latest version and also information about using the code outside

57 + the Linux kernel.

58 +

59 + For userspace, XZ Utils provide a zlib-like compression library

60 + and a gzip-like command line tool. XZ Utils can be downloaded from

61 + <http://tukaani.org/xz/>.

62 +

63 +XZ related components in the kernel

64 +

65 + The xz_dec module provides XZ decompressor with single-call (buffer

66 + to buffer) and multi-call (stateful) APIs. The usage of the xz_dec

67 + module is documented in include/linux/xz.h.

68 +

69 + The xz_dec_test module is for testing xz_dec. xz_dec_test is not

70 + useful unless you are hacking the XZ decompressor. xz_dec_test

71 + allocates a char device major dynamically to which one can write

72 + .xz files from userspace. The decompressed output is thrown away.

73 + Keep an eye on dmesg to see diagnostics printed by xz_dec_test.

74 + See the xz_dec_test source code for the details.

75 +

76 + For decompressing the kernel image, initramfs, and initrd, there

77 + is a wrapper function in lib/decompress_unxz.c. Its API is the

78 + same as in other decompress_*.c files, which is defined in

79 + include/linux/decompress/generic.h.

80 +

81 + scripts/xz_wrap.sh is a wrapper for the xz command line tool found

82 + from XZ Utils. The wrapper sets compression options to values suitable

83 + for compressing the kernel image.

84 +

85 + For kernel makefiles, two commands are provided for use with

86 + $(call if_needed). The kernel image should be compressed with

87 + $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2

88 + dictionary. It will also append a four-byte trailer containing the

89 + uncompressed size of the file, which is needed by the boot code.

90 + Other things should be compressed with $(call if_needed,xzmisc)

91 + which will use no BCJ filter and 1 MiB LZMA2 dictionary.

92 +

93 +Notes on compression options

94 +

95 + Since the XZ Embedded supports only streams with no integrity check or

96 + CRC32, make sure that you don't use some other integrity check type

97 + when encoding files that are supposed to be decoded by the kernel. With

98 + liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32

99 + when encoding. With the xz command line tool, use --check=none or

100 + --check=crc32.

101 +

102 + Using CRC32 is strongly recommended unless there is some other layer

103 + which will verify the integrity of the uncompressed data anyway.

104 + Double checking the integrity would probably be waste of CPU cycles.

105 + Note that the headers will always have a CRC32 which will be validated

106 + by the decoder; you can only change the integrity check type (or

107 + disable it) for the actual uncompressed data.

108 +

109 + In userspace, LZMA2 is typically used with dictionary sizes of several

110 + megabytes. The decoder needs to have the dictionary in RAM, thus big

111 + dictionaries cannot be used for files that are intended to be decoded

112 + by the kernel. 1 MiB is probably the maximum reasonable dictionary

113 + size for in-kernel use (maybe more is OK for initramfs). The presets

114 + in XZ Utils may not be optimal when creating files for the kernel,

115 + so don't hesitate to use custom settings. Example:

116 +

117 + xz --check=crc32 --lzma2=dict=512KiB inputfile

118 +

119 + An exception to above dictionary size limitation is when the decoder

120 + is used in single-call mode. Decompressing the kernel itself is an

121 + example of this situation. In single-call mode, the memory usage

122 + doesn't depend on the dictionary size, and it is perfectly fine to

123 + use a big dictionary: for maximum compression, the dictionary should

124 + be at least as big as the uncompressed data itself.

125 +

126 +Future plans

127 +

128 + Creating a limited XZ encoder may be considered if people think it is

129 + useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at

130 + the fastest settings, so it isn't clear if LZMA2 encoder is wanted

131 + into the kernel.

132 +

133 + Support for limited random-access reading is planned for the

134 + decompression code. I don't know if it could have any use in the

135 + kernel, but I know that it would be useful in some embedded projects

136 + outside the Linux kernel.

137 +

138 +Conformance to the .xz file format specification

139 +

140 + There are a couple of corner cases where things have been simplified

141 + at expense of detecting errors as early as possible. These should not

142 + matter in practice all, since they don't cause security issues. But

143 + it is good to know this if testing the code e.g. with the test files

144 + from XZ Utils.

145 +

146 +Reporting bugs

147 +

148 + Before reporting a bug, please check that it's not fixed already

149 + at upstream. See <http://tukaani.org/xz/embedded.html> to get the

150 + latest code.

151 +

152 + Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on

153 + Freenode and talk to Larhzu. I don't actively read LKML or other

154 + kernel-related mailing lists, so if there's something I should know,

155 + you should email to me personally or use IRC.

156 +

157 + Don't bother Igor Pavlov with questions about the XZ implementation

158 + in the kernel or about XZ Utils. While these two implementations

159 + include essential code that is directly based on Igor Pavlov's code,

160 + these implementations aren't maintained nor supported by him.

161 +

162 diff --git a/include/linux/xz.h b/include/linux/xz.h

163 new file mode 100644

164 index 0000000..64cffa6

165 --- /dev/null

166 +++ b/include/linux/xz.h

167 @@ -0,0 +1,264 @@

168 +/*

169 + * XZ decompressor

170 + *

171 + * Authors: Lasse Collin <lasse.collin@tukaani.org>

172 + * Igor Pavlov <http://7-zip.org/>

173 + *

174 + * This file has been put into the public domain.

175 + * You can do whatever you want with this file.

176 + */

177 +

178 +#ifndef XZ_H

179 +#define XZ_H

180 +

181 +#ifdef __KERNEL__

182 +# include <linux/stddef.h>

183 +# include <linux/types.h>

184 +#else

185 +# include <stddef.h>

186 +# include <stdint.h>

187 +#endif

188 +

189 +/* In Linux, this is used to make extern functions static when needed. */

190 +#ifndef XZ_EXTERN

191 +# define XZ_EXTERN extern

192 +#endif

193 +

194 +/**

195 + * enum xz_mode - Operation mode

196 + *

197 + * @XZ_SINGLE: Single-call mode. This uses less RAM than

198 + * than multi-call modes, because the LZMA2

199 + * dictionary doesn't need to be allocated as

200 + * part of the decoder state. All required data

201 + * structures are allocated at initialization,

202 + * so xz_dec_run() cannot return XZ_MEM_ERROR.

203 + * @XZ_PREALLOC: Multi-call mode with preallocated LZMA2

204 + * dictionary buffer. All data structures are

205 + * allocated at initialization, so xz_dec_run()

206 + * cannot return XZ_MEM_ERROR.

207 + * @XZ_DYNALLOC: Multi-call mode. The LZMA2 dictionary is

208 + * allocated once the required size has been

209 + * parsed from the stream headers. If the

210 + * allocation fails, xz_dec_run() will return

211 + * XZ_MEM_ERROR.

212 + *

213 + * It is possible to enable support only for a subset of the above

214 + * modes at compile time by defining XZ_DEC_SINGLE, XZ_DEC_PREALLOC,

215 + * or XZ_DEC_DYNALLOC. The xz_dec kernel module is always compiled

216 + * with support for all operation modes, but the preboot code may

217 + * be built with fewer features to minimize code size.

218 + */

219 +enum xz_mode {

220 + XZ_SINGLE,

221 + XZ_PREALLOC,

222 + XZ_DYNALLOC

223 +};

224 +

225 +/**

226 + * enum xz_ret - Return codes

227 + * @XZ_OK: Everything is OK so far. More input or more

228 + * output space is required to continue. This

229 + * return code is possible only in multi-call mode

230 + * (XZ_PREALLOC or XZ_DYNALLOC).

231 + * @XZ_STREAM_END: Operation finished successfully.

232 + * @XZ_UNSUPPORTED_CHECK: Integrity check type is not supported. Decoding

233 + * is still possible in multi-call mode by simply

234 + * calling xz_dec_run() again.

235 + * Note that this return value is used only if

236 + * XZ_DEC_ANY_CHECK was defined at build time,

237 + * which is not used in the kernel. Unsupported

238 + * check types return XZ_OPTIONS_ERROR if

239 + * XZ_DEC_ANY_CHECK was not defined at build time.

240 + * @XZ_MEM_ERROR: Allocating memory failed. This return code is

241 + * possible only if the decoder was initialized

242 + * with XZ_DYNALLOC. The amount of memory that was

243 + * tried to be allocated was no more than the

244 + * dict_max argument given to xz_dec_init().

245 + * @XZ_MEMLIMIT_ERROR: A bigger LZMA2 dictionary would be needed than

246 + * allowed by the dict_max argument given to

247 + * xz_dec_init(). This return value is possible

248 + * only in multi-call mode (XZ_PREALLOC or

249 + * XZ_DYNALLOC); the single-call mode (XZ_SINGLE)

250 + * ignores the dict_max argument.

251 + * @XZ_FORMAT_ERROR: File format was not recognized (wrong magic

252 + * bytes).

253 + * @XZ_OPTIONS_ERROR: This implementation doesn't support the requested

254 + * compression options. In the decoder this means

255 + * that the header CRC32 matches, but the header

256 + * itself specifies something that we don't support.

257 + * @XZ_DATA_ERROR: Compressed data is corrupt.

258 + * @XZ_BUF_ERROR: Cannot make any progress. Details are slightly

259 + * different between multi-call and single-call

260 + * mode; more information below.

261 + *

262 + * In multi-call mode, XZ_BUF_ERROR is returned when two consecutive calls

263 + * to XZ code cannot consume any input and cannot produce any new output.

264 + * This happens when there is no new input available, or the output buffer

265 + * is full while at least one output byte is still pending. Assuming your

266 + * code is not buggy, you can get this error only when decoding a compressed

267 + * stream that is truncated or otherwise corrupt.

268 + *

269 + * In single-call mode, XZ_BUF_ERROR is returned only when the output buffer

270 + * is too small or the compressed input is corrupt in a way that makes the

271 + * decoder produce more output than the caller expected. When it is

272 + * (relatively) clear that the compressed input is truncated, XZ_DATA_ERROR

273 + * is used instead of XZ_BUF_ERROR.

274 + */

275 +enum xz_ret {

276 + XZ_OK,

277 + XZ_STREAM_END,

278 + XZ_UNSUPPORTED_CHECK,

279 + XZ_MEM_ERROR,

280 + XZ_MEMLIMIT_ERROR,

281 + XZ_FORMAT_ERROR,

282 + XZ_OPTIONS_ERROR,

283 + XZ_DATA_ERROR,

284 + XZ_BUF_ERROR

285 +};

286 +

287 +/**

288 + * struct xz_buf - Passing input and output buffers to XZ code

289 + * @in: Beginning of the input buffer. This may be NULL if and only

290 + * if in_pos is equal to in_size.

291 + * @in_pos: Current position in the input buffer. This must not exceed

292 + * in_size.

293 + * @in_size: Size of the input buffer

294 + * @out: Beginning of the output buffer. This may be NULL if and only

295 + * if out_pos is equal to out_size.

296 + * @out_pos: Current position in the output buffer. This must not exceed

297 + * out_size.

298 + * @out_size: Size of the output buffer

299 + *

300 + * Only the contents of the output buffer from out[out_pos] onward, and

301 + * the variables in_pos and out_pos are modified by the XZ code.

302 + */

303 +struct xz_buf {

304 + const uint8_t *in;

305 + size_t in_pos;

306 + size_t in_size;

307 +

308 + uint8_t *out;

309 + size_t out_pos;

310 + size_t out_size;

311 +};

312 +

313 +/**

314 + * struct xz_dec - Opaque type to hold the XZ decoder state

315 + */

316 +struct xz_dec;

317 +

318 +/**

319 + * xz_dec_init() - Allocate and initialize a XZ decoder state

320 + * @mode: Operation mode

321 + * @dict_max: Maximum size of the LZMA2 dictionary (history buffer) for

322 + * multi-call decoding. This is ignored in single-call mode

323 + * (mode == XZ_SINGLE). LZMA2 dictionary is always 2^n bytes

324 + * or 2^n + 2^(n-1) bytes (the latter sizes are less common

325 + * in practice), so other values for dict_max don't make sense.

326 + * In the kernel, dictionary sizes of 64 KiB, 128 KiB, 256 KiB,

327 + * 512 KiB, and 1 MiB are probably the only reasonable values,

328 + * except for kernel and initramfs images where a bigger

329 + * dictionary can be fine and useful.

330 + *

331 + * Single-call mode (XZ_SINGLE): xz_dec_run() decodes the whole stream at

332 + * once. The caller must provide enough output space or the decoding will

333 + * fail. The output space is used as the dictionary buffer, which is why

334 + * there is no need to allocate the dictionary as part of the decoder's

335 + * internal state.

336 + *

337 + * Because the output buffer is used as the workspace, streams encoded using

338 + * a big dictionary are not a problem in single-call mode. It is enough that

339 + * the output buffer is big enough to hold the actual uncompressed data; it

340 + * can be smaller than the dictionary size stored in the stream headers.

341 + *

342 + * Multi-call mode with preallocated dictionary (XZ_PREALLOC): dict_max bytes

343 + * of memory is preallocated for the LZMA2 dictionary. This way there is no

344 + * risk that xz_dec_run() could run out of memory, since xz_dec_run() will

345 + * never allocate any memory. Instead, if the preallocated dictionary is too

346 + * small for decoding the given input stream, xz_dec_run() will return

347 + * XZ_MEMLIMIT_ERROR. Thus, it is important to know what kind of data will be

348 + * decoded to avoid allocating excessive amount of memory for the dictionary.

349 + *

350 + * Multi-call mode with dynamically allocated dictionary (XZ_DYNALLOC):

351 + * dict_max specifies the maximum allowed dictionary size that xz_dec_run()

352 + * may allocate once it has parsed the dictionary size from the stream

353 + * headers. This way excessive allocations can be avoided while still

354 + * limiting the maximum memory usage to a sane value to prevent running the

355 + * system out of memory when decompressing streams from untrusted sources.

356 + *

357 + * On success, xz_dec_init() returns a pointer to struct xz_dec, which is

358 + * ready to be used with xz_dec_run(). If memory allocation fails,

359 + * xz_dec_init() returns NULL.

360 + */

361 +XZ_EXTERN struct xz_dec *xz_dec_init(enum xz_mode mode, uint32_t dict_max);

362 +

363 +/**

364 + * xz_dec_run() - Run the XZ decoder

365 + * @s: Decoder state allocated using xz_dec_init()

366 + * @b: Input and output buffers

367 + *

368 + * The possible return values depend on build options and operation mode.

369 + * See enum xz_ret for details.

370 + *

371 + * Note that if an error occurs in single-call mode (return value is not

372 + * XZ_STREAM_END), b->in_pos and b->out_pos are not modified and the

373 + * contents of the output buffer from b->out[b->out_pos] onward are

374 + * undefined. This is true even after XZ_BUF_ERROR, because with some filter

375 + * chains, there may be a second pass over the output buffer, and this pass

376 + * cannot be properly done if the output buffer is truncated. Thus, you

377 + * cannot give the single-call decoder a too small buffer and then expect to

378 + * get that amount valid data from the beginning of the stream. You must use

379 + * the multi-call decoder if you don't want to uncompress the whole stream.

380 + */

381 +XZ_EXTERN enum xz_ret xz_dec_run(struct xz_dec *s, struct xz_buf *b);

382 +

383 +/**

384 + * xz_dec_reset() - Reset an already allocated decoder state

385 + * @s: Decoder state allocated using xz_dec_init()

386 + *

387 + * This function can be used to reset the multi-call decoder state without

388 + * freeing and reallocating memory with xz_dec_end() and xz_dec_init().

389 + *

390 + * In single-call mode, xz_dec_reset() is always called in the beginning of

391 + * xz_dec_run(). Thus, explicit call to xz_dec_reset() is useful only in

392 + * multi-call mode.

393 + */

394 +XZ_EXTERN void xz_dec_reset(struct xz_dec *s);

395 +

396 +/**

397 + * xz_dec_end() - Free the memory allocated for the decoder state

398 + * @s: Decoder state allocated using xz_dec_init(). If s is NULL,

399 + * this function does nothing.

400 + */

401 +XZ_EXTERN void xz_dec_end(struct xz_dec *s);

402 +

403 +/*

404 + * Standalone build (userspace build or in-kernel build for boot time use)

405 + * needs a CRC32 implementation. For normal in-kernel use, kernel's own

406 + * CRC32 module is used instead, and users of this module don't need to

407 + * care about the functions below.

408 + */

409 +#ifndef XZ_INTERNAL_CRC32

410 +# ifdef __KERNEL__

411 +# define XZ_INTERNAL_CRC32 0

412 +# else

413 +# define XZ_INTERNAL_CRC32 1

414 +# endif

415 +#endif

416 +

417 +#if XZ_INTERNAL_CRC32

418 +/*

419 + * This must be called before any other xz_* function to initialize

420 + * the CRC32 lookup table.

421 + */

422 +XZ_EXTERN void xz_crc32_init(void);

423 +

424 +/*

425 + * Update CRC32 value using the polynomial from IEEE-802.3. To start a new

426 + * calculation, the third argument must be zero. To continue the calculation,

427 + * the previously returned value is passed as the third argument.

428 + */

429 +XZ_EXTERN uint32_t xz_crc32(const uint8_t *buf, size_t size, uint32_t crc);

430 +#endif

431 +#endif

432 diff --git a/lib/Kconfig b/lib/Kconfig

433 index fa9bf2c..6090314 100644

434 --- a/lib/Kconfig

435 +++ b/lib/Kconfig

436 @@ -106,6 +106,8 @@ config LZO_COMPRESS

437 config LZO_DECOMPRESS

438 tristate

439

440 +source "lib/xz/Kconfig"

441 +

442 #

443 # These all provide a common interface (hence the apparent duplication with

444 # ZLIB_INFLATE; DECOMPRESS_GZIP is just a wrapper.)

445 diff --git a/lib/Makefile b/lib/Makefile

446 index e6a3763..f2f98dd 100644

447 --- a/lib/Makefile

448 +++ b/lib/Makefile

449 @@ -69,6 +69,7 @@ obj-$(CONFIG_ZLIB_DEFLATE) += zlib_deflate/

450 obj-$(CONFIG_REED_SOLOMON) += reed_solomon/

451 obj-$(CONFIG_LZO_COMPRESS) += lzo/

452 obj-$(CONFIG_LZO_DECOMPRESS) += lzo/

453 +obj-$(CONFIG_XZ_DEC) += xz/

454 obj-$(CONFIG_RAID6_PQ) += raid6/

455

456 lib-$(CONFIG_DECOMPRESS_GZIP) += decompress_inflate.o

457 diff --git a/lib/xz/Kconfig b/lib/xz/Kconfig

458 new file mode 100644

459 index 0000000..e3b6e18

460 --- /dev/null

461 +++ b/lib/xz/Kconfig

462 @@ -0,0 +1,59 @@

463 +config XZ_DEC

464 + tristate "XZ decompression support"

465 + select CRC32

466 + help

467 + LZMA2 compression algorithm and BCJ filters are supported using

468 + the .xz file format as the container. For integrity checking,

469 + CRC32 is supported. See Documentation/xz.txt for more information.

470 +

471 +config XZ_DEC_X86

472 + bool "x86 BCJ filter decoder" if EMBEDDED

473 + default y

474 + depends on XZ_DEC

475 + select XZ_DEC_BCJ

476 +

477 +config XZ_DEC_POWERPC

478 + bool "PowerPC BCJ filter decoder" if EMBEDDED

479 + default y

480 + depends on XZ_DEC

481 + select XZ_DEC_BCJ

482 +

483 +config XZ_DEC_IA64

484 + bool "IA-64 BCJ filter decoder" if EMBEDDED

485 + default y

486 + depends on XZ_DEC

487 + select XZ_DEC_BCJ

488 +

489 +config XZ_DEC_ARM

490 + bool "ARM BCJ filter decoder" if EMBEDDED

491 + default y

492 + depends on XZ_DEC

493 + select XZ_DEC_BCJ

494 +

495 +config XZ_DEC_ARMTHUMB

496 + bool "ARM-Thumb BCJ filter decoder" if EMBEDDED

497 + default y

498 + depends on XZ_DEC

499 + select XZ_DEC_BCJ

500 +

501 +config XZ_DEC_SPARC

502 + bool "SPARC BCJ filter decoder" if EMBEDDED

503 + default y

504 + depends on XZ_DEC

505 + select XZ_DEC_BCJ

506 +

507 +config XZ_DEC_BCJ

508 + bool

509 + default n

510 +

511 +config XZ_DEC_TEST

512 + tristate "XZ decompressor tester"

513 + default n

514 + depends on XZ_DEC

515 + help

516 + This allows passing .xz files to the in-kernel XZ decoder via

517 + a character special file. It calculates CRC32 of the decompressed

518 + data and writes diagnostics to the system log.

519 +

520 + Unless you are developing the XZ decoder, you don't need this

521 + and should say N.

522 diff --git a/lib/xz/Makefile b/lib/xz/Makefile

523 new file mode 100644

524 index 0000000..a7fa769

525 --- /dev/null

526 +++ b/lib/xz/Makefile

527 @@ -0,0 +1,5 @@

528 +obj-$(CONFIG_XZ_DEC) += xz_dec.o

529 +xz_dec-y := xz_dec_syms.o xz_dec_stream.o xz_dec_lzma2.o

530 +xz_dec-$(CONFIG_XZ_DEC_BCJ) += xz_dec_bcj.o

531 +

532 +obj-$(CONFIG_XZ_DEC_TEST) += xz_dec_test.o

533 diff --git a/lib/xz/xz_crc32.c b/lib/xz/xz_crc32.c

534 new file mode 100644

535 index 0000000..34532d1

536 --- /dev/null

537 +++ b/lib/xz/xz_crc32.c

538 @@ -0,0 +1,59 @@

539 +/*

540 + * CRC32 using the polynomial from IEEE-802.3

541 + *

542 + * Authors: Lasse Collin <lasse.collin@tukaani.org>

543 + * Igor Pavlov <http://7-zip.org/>

544 + *

545 + * This file has been put into the public domain.

546 + * You can do whatever you want with this file.

547 + */

548 +

549 +/*

550 + * This is not the fastest implementation, but it is pretty compact.

551 + * The fastest versions of xz_crc32() on modern CPUs without hardware

552 + * accelerated CRC instruction are 3-5 times as fast as this version,

553 + * but they are bigger and use more memory for the lookup table.

554 + */

555 +

556 +#include "xz_private.h"

557 +

558 +/*

559 + * STATIC_RW_DATA is used in the pre-boot environment on some architectures.

560 + * See <linux/decompress/mm.h> for details.

561 + */

562 +#ifndef STATIC_RW_DATA

563 +# define STATIC_RW_DATA static

564 +#endif

565 +

566 +STATIC_RW_DATA uint32_t xz_crc32_table[256];

567 +

568 +XZ_EXTERN void xz_crc32_init(void)

569 +{

570 + const uint32_t poly = 0xEDB88320;

571 +

572 + uint32_t i;

573 + uint32_t j;

574 + uint32_t r;

575 +

576 + for (i = 0; i < 256; ++i) {

577 + r = i;

578 + for (j = 0; j < 8; ++j)

579 + r = (r >> 1) ^ (poly & ~((r & 1) - 1));

580 +

581 + xz_crc32_table[i] = r;

582 + }

583 +

584 + return;

585 +}

586 +

587 +XZ_EXTERN uint32_t xz_crc32(const uint8_t *buf, size_t size, uint32_t crc)

588 +{

589 + crc = ~crc;

590 +

591 + while (size != 0) {

592 + crc = xz_crc32_table[*buf++ ^ (crc & 0xFF)] ^ (crc >> 8);

593 + --size;

594 + }

595 +

596 + return ~crc;

597 +}

598 diff --git a/lib/xz/xz_dec_bcj.c b/lib/xz/xz_dec_bcj.c

599 new file mode 100644

600 index 0000000..e51e255

601 --- /dev/null

602 +++ b/lib/xz/xz_dec_bcj.c

603 @@ -0,0 +1,561 @@

604 +/*

605 + * Branch/Call/Jump (BCJ) filter decoders

606 + *

607 + * Authors: Lasse Collin <lasse.collin@tukaani.org>

608 + * Igor Pavlov <http://7-zip.org/>

609 + *

610 + * This file has been put into the public domain.

611 + * You can do whatever you want with this file.

612 + */

613 +

614 +#include "xz_private.h"

615 +

616 +/*

617 + * The rest of the file is inside this ifdef. It makes things a little more

618 + * convenient when building without support for any BCJ filters.

619 + */

620 +#ifdef XZ_DEC_BCJ

621 +

622 +struct xz_dec_bcj {

623 + /* Type of the BCJ filter being used */

624 + enum {

625 + BCJ_X86 = 4, /* x86 or x86-64 */

626 + BCJ_POWERPC = 5, /* Big endian only */

627 + BCJ_IA64 = 6, /* Big or little endian */

628 + BCJ_ARM = 7, /* Little endian only */

629 + BCJ_ARMTHUMB = 8, /* Little endian only */

630 + BCJ_SPARC = 9 /* Big or little endian */

631 + } type;

632 +

633 + /*

634 + * Return value of the next filter in the chain. We need to preserve

635 + * this information across calls, because we must not call the next

636 + * filter anymore once it has returned XZ_STREAM_END.

637 + */

638 + enum xz_ret ret;

639 +

640 + /* True if we are operating in single-call mode. */

641 + bool single_call;

642 +

643 + /*

644 + * Absolute position relative to the beginning of the uncompressed

645 + * data (in a single .xz Block). We care only about the lowest 32

646 + * bits so this doesn't need to be uint64_t even with big files.

647 + */

648 + uint32_t pos;

649 +

650 + /* x86 filter state */

651 + uint32_t x86_prev_mask;

652 +

653 + /* Temporary space to hold the variables from struct xz_buf */

654 + uint8_t *out;

655 + size_t out_pos;

656 + size_t out_size;

657 +

658 + struct {

659 + /* Amount of already filtered data in the beginning of buf */

660 + size_t filtered;

661 +

662 + /* Total amount of data currently stored in buf */

663 + size_t size;

664 +

665 + /*

666 + * Buffer to hold a mix of filtered and unfiltered data. This

667 + * needs to be big enough to hold Alignment + 2 * Look-ahead:

668 + *

669 + * Type Alignment Look-ahead

670 + * x86 1 4

671 + * PowerPC 4 0

672 + * IA-64 16 0

673 + * ARM 4 0

674 + * ARM-Thumb 2 2

675 + * SPARC 4 0

676 + */

677 + uint8_t buf[16];

678 + } temp;

679 +};

680 +

681 +#ifdef XZ_DEC_X86

682 +/*

683 + * This is used to test the most significant byte of a memory address

684 + * in an x86 instruction.

685 + */

686 +static inline int bcj_x86_test_msbyte(uint8_t b)

687 +{

688 + return b == 0x00 || b == 0xFF;

689 +}

690 +

691 +static size_t bcj_x86(struct xz_dec_bcj *s, uint8_t *buf, size_t size)

692 +{

693 + static const bool mask_to_allowed_status[8]

694 + = { true, true, true, false, true, false, false, false };

695 +

696 + static const uint8_t mask_to_bit_num[8] = { 0, 1, 2, 2, 3, 3, 3, 3 };

697 +

698 + size_t i;

699 + size_t prev_pos = (size_t)-1;

700 + uint32_t prev_mask = s->x86_prev_mask;

701 + uint32_t src;

702 + uint32_t dest;

703 + uint32_t j;

704 + uint8_t b;

705 +

706 + if (size <= 4)

707 + return 0;

708 +

709 + size -= 4;

710 + for (i = 0; i < size; ++i) {

711 + if ((buf[i] & 0xFE) != 0xE8)

712 + continue;

713 +

714 + prev_pos = i - prev_pos;

715 + if (prev_pos > 3) {

716 + prev_mask = 0;

717 + } else {

718 + prev_mask = (prev_mask << (prev_pos - 1)) & 7;

719 + if (prev_mask != 0) {

720 + b = buf[i + 4 - mask_to_bit_num[prev_mask]];

721 + if (!mask_to_allowed_status[prev_mask]

722 + || bcj_x86_test_msbyte(b)) {

723 + prev_pos = i;

724 + prev_mask = (prev_mask << 1) | 1;

725 + continue;

726 + }

727 + }

728 + }

729 +

730 + prev_pos = i;

731 +

732 + if (bcj_x86_test_msbyte(buf[i + 4])) {

733 + src = get_unaligned_le32(buf + i + 1);

734 + while (true) {

735 + dest = src - (s->pos + (uint32_t)i + 5);

736 + if (prev_mask == 0)

737 + break;

738 +

739 + j = mask_to_bit_num[prev_mask] * 8;

740 + b = (uint8_t)(dest >> (24 - j));

741 + if (!bcj_x86_test_msbyte(b))

742 + break;

743 +

744 + src = dest ^ (((uint32_t)1 << (32 - j)) - 1);

745 + }

746 +

747 + dest &= 0x01FFFFFF;

748 + dest |= (uint32_t)0 - (dest & 0x01000000);

749 + put_unaligned_le32(dest, buf + i + 1);

750 + i += 4;

751 + } else {

752 + prev_mask = (prev_mask << 1) | 1;

753 + }

754 + }

755 +

756 + prev_pos = i - prev_pos;

757 + s->x86_prev_mask = prev_pos > 3 ? 0 : prev_mask << (prev_pos - 1);

758 + return i;

759 +}

760 +#endif

761 +

762 +#ifdef XZ_DEC_POWERPC

763 +static size_t bcj_powerpc(struct xz_dec_bcj *s, uint8_t *buf, size_t size)

764 +{

765 + size_t i;

766 + uint32_t instr;

767 +

768 + for (i = 0; i + 4 <= size; i += 4) {

769 + instr = get_unaligned_be32(buf + i);

770 + if ((instr & 0xFC000003) == 0x48000001) {

771 + instr &= 0x03FFFFFC;

772 + instr -= s->pos + (uint32_t)i;

773 + instr &= 0x03FFFFFC;

774 + instr |= 0x48000001;

775 + put_unaligned_be32(instr, buf + i);

776 + }

777 + }

778 +

779 + return i;

780 +}

781 +#endif

782 +

783 +#ifdef XZ_DEC_IA64

784 +static size_t bcj_ia64(struct xz_dec_bcj *s, uint8_t *buf, size_t size)

785 +{

786 + static const uint8_t branch_table[32] = {

787 + 0, 0, 0, 0, 0, 0, 0, 0,

788 + 0, 0, 0, 0, 0, 0, 0, 0,

789 + 4, 4, 6, 6, 0, 0, 7, 7,

790 + 4, 4, 0, 0, 4, 4, 0, 0

791 + };

792 +

793 + /*

794 + * The local variables take a little bit stack space, but it's less

795 + * than what LZMA2 decoder takes, so it doesn't make sense to reduce

796 + * stack usage here without doing that for the LZMA2 decoder too.

797 + */

798 +

799 + /* Loop counters */

800 + size_t i;

801 + size_t j;

802 +

803 + /* Instruction slot (0, 1, or 2) in the 128-bit instruction word */

804 + uint32_t slot;

805 +

806 + /* Bitwise offset of the instruction indicated by slot */

807 + uint32_t bit_pos;

808 +

809 + /* bit_pos split into byte and bit parts */

810 + uint32_t byte_pos;

811 + uint32_t bit_res;

812 +

813 + /* Address part of an instruction */

814 + uint32_t addr;

815 +

816 + /* Mask used to detect which instructions to convert */

817 + uint32_t mask;

818 +

819 + /* 41-bit instruction stored somewhere in the lowest 48 bits */

820 + uint64_t instr;

821 +

822 + /* Instruction normalized with bit_res for easier manipulation */

823 + uint64_t norm;

824 +

825 + for (i = 0; i + 16 <= size; i += 16) {

826 + mask = branch_table[buf[i] & 0x1F];

827 + for (slot = 0, bit_pos = 5; slot < 3; ++slot, bit_pos += 41) {

828 + if (((mask >> slot) & 1) == 0)

829 + continue;

830 +

831 + byte_pos = bit_pos >> 3;

832 + bit_res = bit_pos & 7;

833 + instr = 0;

834 + for (j = 0; j < 6; ++j)

835 + instr |= (uint64_t)(buf[i + j + byte_pos])

836 + << (8 * j);

837 +

838 + norm = instr >> bit_res;

839 +

840 + if (((norm >> 37) & 0x0F) == 0x05

841 + && ((norm >> 9) & 0x07) == 0) {

842 + addr = (norm >> 13) & 0x0FFFFF;

843 + addr |= ((uint32_t)(norm >> 36) & 1) << 20;

844 + addr <<= 4;

845 + addr -= s->pos + (uint32_t)i;

846 + addr >>= 4;

847 +

848 + norm &= ~((uint64_t)0x8FFFFF << 13);

849 + norm |= (uint64_t)(addr & 0x0FFFFF) << 13;

850 + norm |= (uint64_t)(addr & 0x100000)

851 + << (36 - 20);

852 +

853 + instr &= (1 << bit_res) - 1;

854 + instr |= norm << bit_res;

855 +

856 + for (j = 0; j < 6; j++)

857 + buf[i + j + byte_pos]

858 + = (uint8_t)(instr >> (8 * j));

859 + }

860 + }

861 + }

862 +

863 + return i;

864 +}

865 +#endif

866 +

867 +#ifdef XZ_DEC_ARM

868 +static size_t bcj_arm(struct xz_dec_bcj *s, uint8_t *buf, size_t size)

869 +{

870 + size_t i;

871 + uint32_t addr;

872 +

873 + for (i = 0; i + 4 <= size; i += 4) {

874 + if (buf[i + 3] == 0xEB) {

875 + addr = (uint32_t)buf[i] | ((uint32_t)buf[i + 1] << 8)

876 + | ((uint32_t)buf[i + 2] << 16);

877 + addr <<= 2;

878 + addr -= s->pos + (uint32_t)i + 8;

879 + addr >>= 2;

880 + buf[i] = (uint8_t)addr;

881 + buf[i + 1] = (uint8_t)(addr >> 8);

882 + buf[i + 2] = (uint8_t)(addr >> 16);

883 + }

884 + }

885 +

886 + return i;

887 +}

888 +#endif

889 +

890 +#ifdef XZ_DEC_ARMTHUMB

891 +static size_t bcj_armthumb(struct xz_dec_bcj *s, uint8_t *buf, size_t size)

892 +{

893 + size_t i;

894 + uint32_t addr;

895 +

896 + for (i = 0; i + 4 <= size; i += 2) {

897 + if ((buf[i + 1] & 0xF8) == 0xF0

898 + && (buf[i + 3] & 0xF8) == 0xF8) {

899 + addr = (((uint32_t)buf[i + 1] & 0x07) << 19)

900 + | ((uint32_t)buf[i] << 11)

901 + | (((uint32_t)buf[i + 3] & 0x07) << 8)

902 + | (uint32_t)buf[i + 2];

903 + addr <<= 1;

904 + addr -= s->pos + (uint32_t)i + 4;

905 + addr >>= 1;

906 + buf[i + 1] = (uint8_t)(0xF0 | ((addr >> 19) & 0x07));

907 + buf[i] = (uint8_t)(addr >> 11);

908 + buf[i + 3] = (uint8_t)(0xF8 | ((addr >> 8) & 0x07));

909 + buf[i + 2] = (uint8_t)addr;

910 + i += 2;

911 + }

912 + }

913 +

914 + return i;

915 +}

916 +#endif

917 +

918 +#ifdef XZ_DEC_SPARC

919 +static size_t bcj_sparc(struct xz_dec_bcj *s, uint8_t *buf, size_t size)

920 +{

921 + size_t i;

922 + uint32_t instr;

923 +

924 + for (i = 0; i + 4 <= size; i += 4) {

925 + instr = get_unaligned_be32(buf + i);

926 + if ((instr >> 22) == 0x100 || (instr >> 22) == 0x1FF) {

927 + instr <<= 2;

928 + instr -= s->pos + (uint32_t)i;

929 + instr >>= 2;

930 + instr = ((uint32_t)0x40000000 - (instr & 0x400000))

931 + | 0x40000000 | (instr & 0x3FFFFF);

932 + put_unaligned_be32(instr, buf + i);

933 + }

934 + }

935 +

936 + return i;

937 +}

938 +#endif

939 +

940 +/*

941 + * Apply the selected BCJ filter. Update *pos and s->pos to match the amount

942 + * of data that got filtered.

943 + *

944 + * NOTE: This is implemented as a switch statement to avoid using function

945 + * pointers, which could be problematic in the kernel boot code, which must

946 + * avoid pointers to static data (at least on x86).

947 + */

948 +static void bcj_apply(struct xz_dec_bcj *s,

949 + uint8_t *buf, size_t *pos, size_t size)

950 +{

951 + size_t filtered;

952 +

953 + buf += *pos;

954 + size -= *pos;

955 +

956 + switch (s->type) {

957 +#ifdef XZ_DEC_X86

958 + case BCJ_X86:

959 + filtered = bcj_x86(s, buf, size);

960 + break;

961 +#endif

962 +#ifdef XZ_DEC_POWERPC

963 + case BCJ_POWERPC:

964 + filtered = bcj_powerpc(s, buf, size);

965 + break;

966 +#endif

967 +#ifdef XZ_DEC_IA64

968 + case BCJ_IA64:

969 + filtered = bcj_ia64(s, buf, size);

970 + break;

971 +#endif

972 +#ifdef XZ_DEC_ARM

973 + case BCJ_ARM:

974 + filtered = bcj_arm(s, buf, size);

975 + break;

976 +#endif

977 +#ifdef XZ_DEC_ARMTHUMB

978 + case BCJ_ARMTHUMB:

979 + filtered = bcj_armthumb(s, buf, size);

980 + break;

981 +#endif

982 +#ifdef XZ_DEC_SPARC

983 + case BCJ_SPARC:

984 + filtered = bcj_sparc(s, buf, size);

985 + break;

986 +#endif

987 + default:

988 + /* Never reached but silence compiler warnings. */

989 + filtered = 0;

990 + break;

991 + }

992 +

993 + *pos += filtered;

994 + s->pos += filtered;

995 +}

996 +

997 +/*

998 + * Flush pending filtered data from temp to the output buffer.

999 + * Move the remaining mixture of possibly filtered and unfiltered

1000 + * data to the beginning of temp.

1001 + */

1002 +static void bcj_flush(struct xz_dec_bcj *s, struct xz_buf *b)

1003 +{

1004 + size_t copy_size;

1005 +

1006 + copy_size = min_t(size_t, s->temp.filtered, b->out_size - b->out_pos);

1007 + memcpy(b->out + b->out_pos, s->temp.buf, copy_size);

1008 + b->out_pos += copy_size;

1009 +

1010 + s->temp.filtered -= copy_size;

1011 + s->temp.size -= copy_size;

1012 + memmove(s->temp.buf, s->temp.buf + copy_size, s->temp.size);

1013 +}

1014 +

1015 +/*

1016 + * The BCJ filter functions are primitive in sense that they process the

1017 + * data in chunks of 1-16 bytes. To hide this issue, this function does

1018 + * some buffering.

1019 + */

1020 +XZ_EXTERN enum xz_ret xz_dec_bcj_run(struct xz_dec_bcj *s,

1021 + struct xz_dec_lzma2 *lzma2,

1022 + struct xz_buf *b)

1023 +{

1024 + size_t out_start;

1025 +

1026 + /*

1027 + * Flush pending already filtered data to the output buffer. Return

1028 + * immediatelly if we couldn't flush everything, or if the next

1029 + * filter in the chain had already returned XZ_STREAM_END.

1030 + */

1031 + if (s->temp.filtered > 0) {

1032 + bcj_flush(s, b);

1033 + if (s->temp.filtered > 0)

1034 + return XZ_OK;

1035 +

1036 + if (s->ret == XZ_STREAM_END)

1037 + return XZ_STREAM_END;

1038 + }

1039 +

1040 + /*

1041 + * If we have more output space than what is currently pending in

1042 + * temp, copy the unfiltered data from temp to the output buffer

1043 + * and try to fill the output buffer by decoding more data from the

1044 + * next filter in the chain. Apply the BCJ filter on the new data

1045 + * in the output buffer. If everything cannot be filtered, copy it

1046 + * to temp and rewind the output buffer position accordingly.

1047 + */

1048 + if (s->temp.size < b->out_size - b->out_pos) {

1049 + out_start = b->out_pos;

1050 + memcpy(b->out + b->out_pos, s->temp.buf, s->temp.size);

1051 + b->out_pos += s->temp.size;

1052 +

1053 + s->ret = xz_dec_lzma2_run(lzma2, b);

1054 + if (s->ret != XZ_STREAM_END

1055 + && (s->ret != XZ_OK || s->single_call))

1056 + return s->ret;

1057 +

1058 + bcj_apply(s, b->out, &out_start, b->out_pos);

1059 +

1060 + /*

1061 + * As an exception, if the next filter returned XZ_STREAM_END,

1062 + * we can do that too, since the last few bytes that remain

1063 + * unfiltered are meant to remain unfiltered.

1064 + */

1065 + if (s->ret == XZ_STREAM_END)

1066 + return XZ_STREAM_END;

1067 +

1068 + s->temp.size = b->out_pos - out_start;

1069 + b->out_pos -= s->temp.size;

1070 + memcpy(s->temp.buf, b->out + b->out_pos, s->temp.size);

1071 + }

1072 +

1073 + /*

1074 + * If we have unfiltered data in temp, try to fill by decoding more

1075 + * data from the next filter. Apply the BCJ filter on temp. Then we

1076 + * hopefully can fill the actual output buffer by copying filtered

1077 + * data from temp. A mix of filtered and unfiltered data may be left

1078 + * in temp; it will be taken care on the next call to this function.

1079 + */

1080 + if (s->temp.size > 0) {

1081 + /* Make b->out{,_pos,_size} temporarily point to s->temp. */

1082 + s->out = b->out;

1083 + s->out_pos = b->out_pos;

1084 + s->out_size = b->out_size;

1085 + b->out = s->temp.buf;

1086 + b->out_pos = s->temp.size;

1087 + b->out_size = sizeof(s->temp.buf);

1088 +

1089 + s->ret = xz_dec_lzma2_run(lzma2, b);

1090 +

1091 + s->temp.size = b->out_pos;

1092 + b->out = s->out;

1093 + b->out_pos = s->out_pos;

1094 + b->out_size = s->out_size;

1095 +

1096 + if (s->ret != XZ_OK && s->ret != XZ_STREAM_END)

1097 + return s->ret;

1098 +

1099 + bcj_apply(s, s->temp.buf, &s->temp.filtered, s->temp.size);

1100 +

1101 + /*

1102 + * If the next filter returned XZ_STREAM_END, we mark that

1103 + * everything is filtered, since the last unfiltered bytes

1104 + * of the stream are meant to be left as is.

1105 + */

1106 + if (s->ret == XZ_STREAM_END)

1107 + s->temp.filtered = s->temp.size;

1108 +

1109 + bcj_flush(s, b);

1110 + if (s->temp.filtered > 0)

1111 + return XZ_OK;

1112 + }

1113 +

1114 + return s->ret;

1115 +}

1116 +

1117 +XZ_EXTERN struct xz_dec_bcj *xz_dec_bcj_create(bool single_call)

1118 +{

1119 + struct xz_dec_bcj *s = kmalloc(sizeof(*s), GFP_KERNEL);

1120 + if (s != NULL)

1121 + s->single_call = single_call;

1122 +

1123 + return s;

1124 +}

1125 +

1126 +XZ_EXTERN enum xz_ret xz_dec_bcj_reset(struct xz_dec_bcj *s, uint8_t id)

1127 +{

1128 + switch (id) {

1129 +#ifdef XZ_DEC_X86

1130 + case BCJ_X86:

1131 +#endif

1132 +#ifdef XZ_DEC_POWERPC

1133 + case BCJ_POWERPC:

1134 +#endif

1135 +#ifdef XZ_DEC_IA64

1136 + case BCJ_IA64:

1137 +#endif

1138 +#ifdef XZ_DEC_ARM

1139 + case BCJ_ARM:

1140 +#endif

1141 +#ifdef XZ_DEC_ARMTHUMB

1142 + case BCJ_ARMTHUMB:

1143 +#endif

1144 +#ifdef XZ_DEC_SPARC

1145 + case BCJ_SPARC:

1146 +#endif

1147 + break;

1148 +

1149 + default:

1150 + /* Unsupported Filter ID */

1151 + return XZ_OPTIONS_ERROR;

1152 + }

1153 +

1154 + s->type = id;

1155 + s->ret = XZ_OK;

1156 + s->pos = 0;

1157 + s->x86_prev_mask = 0;

1158 + s->temp.filtered = 0;

1159 + s->temp.size = 0;

1160 +

1161 + return XZ_OK;

1162 +}

1163 +

1164 +#endif

1165 diff --git a/lib/xz/xz_dec_lzma2.c b/lib/xz/xz_dec_lzma2.c

1166 new file mode 100644

1167 index 0000000..ea5fa4f

1168 --- /dev/null

1169 +++ b/lib/xz/xz_dec_lzma2.c

1170 @@ -0,0 +1,1171 @@

1171 +/*

1172 + * LZMA2 decoder

1173 + *

1174 + * Authors: Lasse Collin <lasse.collin@tukaani.org>

1175 + * Igor Pavlov <http://7-zip.org/>

1176 + *

1177 + * This file has been put into the public domain.

1178 + * You can do whatever you want with this file.

1179 + */

1180 +

1181 +#include "xz_private.h"

1182 +#include "xz_lzma2.h"

1183 +

1184 +/*

1185 + * Range decoder initialization eats the first five bytes of each LZMA chunk.

1186 + */

1187 +#define RC_INIT_BYTES 5

1188 +

1189 +/*

1190 + * Minimum number of usable input buffer to safely decode one LZMA symbol.

1191 + * The worst case is that we decode 22 bits using probabilities and 26

1192 + * direct bits. This may decode at maximum of 20 bytes of input. However,

1193 + * lzma_main() does an extra normalization before returning, thus we

1194 + * need to put 21 here.

1195 + */

1196 +#define LZMA_IN_REQUIRED 21

1197 +

1198 +/*

1199 + * Dictionary (history buffer)

1200 + *

1201 + * These are always true:

1202 + * start <= pos <= full <= end

1203 + * pos <= limit <= end

1204 + *

1205 + * In multi-call mode, also these are true:

1206 + * end == size

1207 + * size <= size_max

1208 + * allocated <= size

1209 + *

1210 + * Most of these variables are size_t to support single-call mode,

1211 + * in which the dictionary variables address the actual output

1212 + * buffer directly.

1213 + */

1214 +struct dictionary {

1215 + /* Beginning of the history buffer */

1216 + uint8_t *buf;

1217 +

1218 + /* Old position in buf (before decoding more data) */

1219 + size_t start;

1220 +

1221 + /* Position in buf */

1222 + size_t pos;

1223 +

1224 + /*

1225 + * How full dictionary is. This is used to detect corrupt input that

1226 + * would read beyond the beginning of the uncompressed stream.

1227 + */

1228 + size_t full;

1229 +

1230 + /* Write limit; we don't write to buf[limit] or later bytes. */

1231 + size_t limit;

1232 +

1233 + /*

1234 + * End of the dictionary buffer. In multi-call mode, this is

1235 + * the same as the dictionary size. In single-call mode, this

1236 + * indicates the size of the output buffer.

1237 + */

1238 + size_t end;

1239 +

1240 + /*

1241 + * Size of the dictionary as specified in Block Header. This is used

1242 + * together with "full" to detect corrupt input that would make us

1243 + * read beyond the beginning of the uncompressed stream.

1244 + */

1245 + uint32_t size;

1246 +

1247 + /*

1248 + * Maximum allowed dictionary size in multi-call mode.

1249 + * This is ignored in single-call mode.

1250 + */

1251 + uint32_t size_max;

1252 +

1253 + /*

1254 + * Amount of memory currently allocated for the dictionary.

1255 + * This is used only with XZ_DYNALLOC. (With XZ_PREALLOC,

1256 + * size_max is always the same as the allocated size.)

1257 + */

1258 + uint32_t allocated;

1259 +

1260 + /* Operation mode */

1261 + enum xz_mode mode;

1262 +};

1263 +

1264 +/* Range decoder */

1265 +struct rc_dec {

1266 + uint32_t range;

1267 + uint32_t code;

1268 +

1269 + /*

1270 + * Number of initializing bytes remaining to be read

1271 + * by rc_read_init().

1272 + */

1273 + uint32_t init_bytes_left;

1274 +

1275 + /*

1276 + * Buffer from which we read our input. It can be either

1277 + * temp.buf or the caller-provided input buffer.

1278 + */

1279 + const uint8_t *in;

1280 + size_t in_pos;

1281 + size_t in_limit;

1282 +};

1283 +

1284 +/* Probabilities for a length decoder. */

1285 +struct lzma_len_dec {

1286 + /* Probability of match length being at least 10 */

1287 + uint16_t choice;

1288 +

1289 + /* Probability of match length being at least 18 */

1290 + uint16_t choice2;

1291 +

1292 + /* Probabilities for match lengths 2-9 */

1293 + uint16_t low[POS_STATES_MAX][LEN_LOW_SYMBOLS];

1294 +

1295 + /* Probabilities for match lengths 10-17 */

1296 + uint16_t mid[POS_STATES_MAX][LEN_MID_SYMBOLS];

1297 +

1298 + /* Probabilities for match lengths 18-273 */

1299 + uint16_t high[LEN_HIGH_SYMBOLS];

1300 +};

1301 +

1302 +struct lzma_dec {

1303 + /* Distances of latest four matches */

1304 + uint32_t rep0;

1305 + uint32_t rep1;

1306 + uint32_t rep2;

1307 + uint32_t rep3;

1308 +

1309 + /* Types of the most recently seen LZMA symbols */

1310 + enum lzma_state state;

1311 +

1312 + /*

1313 + * Length of a match. This is updated so that dict_repeat can

1314 + * be called again to finish repeating the whole match.

1315 + */

1316 + uint32_t len;

1317 +

1318 + /*

1319 + * LZMA properties or related bit masks (number of literal

1320 + * context bits, a mask dervied from the number of literal

1321 + * position bits, and a mask dervied from the number

1322 + * position bits)

1323 + */

1324 + uint32_t lc;

1325 + uint32_t literal_pos_mask; /* (1 << lp) - 1 */

1326 + uint32_t pos_mask; /* (1 << pb) - 1 */

1327 +

1328 + /* If 1, it's a match. Otherwise it's a single 8-bit literal. */

1329 + uint16_t is_match[STATES][POS_STATES_MAX];

1330 +

1331 + /* If 1, it's a repeated match. The distance is one of rep0 .. rep3. */

1332 + uint16_t is_rep[STATES];

1333 +

1334 + /*

1335 + * If 0, distance of a repeated match is rep0.

1336 + * Otherwise check is_rep1.

1337 + */

1338 + uint16_t is_rep0[STATES];

1339 +

1340 + /*

1341 + * If 0, distance of a repeated match is rep1.

1342 + * Otherwise check is_rep2.

1343 + */

1344 + uint16_t is_rep1[STATES];

1345 +

1346 + /* If 0, distance of a repeated match is rep2. Otherwise it is rep3. */

1347 + uint16_t is_rep2[STATES];

1348 +

1349 + /*

1350 + * If 1, the repeated match has length of one byte. Otherwise

1351 + * the length is decoded from rep_len_decoder.

1352 + */

1353 + uint16_t is_rep0_long[STATES][POS_STATES_MAX];

1354 +

1355 + /*

1356 + * Probability tree for the highest two bits of the match

1357 + * distance. There is a separate probability tree for match

1358 + * lengths of 2 (i.e. MATCH_LEN_MIN), 3, 4, and [5, 273].

1359 + */

1360 + uint16_t dist_slot[DIST_STATES][DIST_SLOTS];

1361 +

1362 + /*

1363 + * Probility trees for additional bits for match distance

1364 + * when the distance is in the range [4, 127].

1365 + */

1366 + uint16_t dist_special[FULL_DISTANCES - DIST_MODEL_END];

1367 +

1368 + /*

1369 + * Probability tree for the lowest four bits of a match

1370 + * distance that is equal to or greater than 128.

1371 + */

1372 + uint16_t dist_align[ALIGN_SIZE];

1373 +

1374 + /* Length of a normal match */

1375 + struct lzma_len_dec match_len_dec;

1376 +

1377 + /* Length of a repeated match */

1378 + struct lzma_len_dec rep_len_dec;

1379 +

1380 + /* Probabilities of literals */

1381 + uint16_t literal[LITERAL_CODERS_MAX][LITERAL_CODER_SIZE];

1382 +};

1383 +

1384 +struct lzma2_dec {

1385 + /* Position in xz_dec_lzma2_run(). */

1386 + enum lzma2_seq {

1387 + SEQ_CONTROL,

1388 + SEQ_UNCOMPRESSED_1,

1389 + SEQ_UNCOMPRESSED_2,

1390 + SEQ_COMPRESSED_0,

1391 + SEQ_COMPRESSED_1,

1392 + SEQ_PROPERTIES,

1393 + SEQ_LZMA_PREPARE,

1394 + SEQ_LZMA_RUN,

1395 + SEQ_COPY

1396 + } sequence;

1397 +

1398 + /* Next position after decoding the compressed size of the chunk. */

1399 + enum lzma2_seq next_sequence;

1400 +

1401 + /* Uncompressed size of LZMA chunk (2 MiB at maximum) */

1402 + uint32_t uncompressed;

1403 +

1404 + /*

1405 + * Compressed size of LZMA chunk or compressed/uncompressed

1406 + * size of uncompressed chunk (64 KiB at maximum)

1407 + */

1408 + uint32_t compressed;

1409 +

1410 + /*

1411 + * True if dictionary reset is needed. This is false before

1412 + * the first chunk (LZMA or uncompressed).

1413 + */

1414 + bool need_dict_reset;

1415 +

1416 + /*

1417 + * True if new LZMA properties are needed. This is false

1418 + * before the first LZMA chunk.

1419 + */

1420 + bool need_props;

1421 +};

1422 +

1423 +struct xz_dec_lzma2 {

1424 + /*

1425 + * The order below is important on x86 to reduce code size and

1426 + * it shouldn't hurt on other platforms. Everything up to and

1427 + * including lzma.pos_mask are in the first 128 bytes on x86-32,

1428 + * which allows using smaller instructions to access those

1429 + * variables. On x86-64, fewer variables fit into the first 128

1430 + * bytes, but this is still the best order without sacrificing

1431 + * the readability by splitting the structures.

1432 + */

1433 + struct rc_dec rc;

1434 + struct dictionary dict;

1435 + struct lzma2_dec lzma2;

1436 + struct lzma_dec lzma;

1437 +

1438 + /*

1439 + * Temporary buffer which holds small number of input bytes between

1440 + * decoder calls. See lzma2_lzma() for details.

1441 + */

1442 + struct {

1443 + uint32_t size;

1444 + uint8_t buf[3 * LZMA_IN_REQUIRED];

1445 + } temp;

1446 +};

1447 +

1448 +/**************

1449 + * Dictionary *

1450 + **************/

1451 +

1452 +/*

1453 + * Reset the dictionary state. When in single-call mode, set up the beginning

1454 + * of the dictionary to point to the actual output buffer.

1455 + */

1456 +static void dict_reset(struct dictionary *dict, struct xz_buf *b)

1457 +{

1458 + if (DEC_IS_SINGLE(dict->mode)) {

1459 + dict->buf = b->out + b->out_pos;

1460 + dict->end = b->out_size - b->out_pos;

1461 + }

1462 +

1463 + dict->start = 0;

1464 + dict->pos = 0;

1465 + dict->limit = 0;

1466 + dict->full = 0;

1467 +}

1468 +

1469 +/* Set dictionary write limit */

1470 +static void dict_limit(struct dictionary *dict, size_t out_max)

1471 +{

1472 + if (dict->end - dict->pos <= out_max)

1473 + dict->limit = dict->end;

1474 + else

1475 + dict->limit = dict->pos + out_max;

1476 +}

1477 +

1478 +/* Return true if at least one byte can be written into the dictionary. */

1479 +static inline bool dict_has_space(const struct dictionary *dict)

1480 +{

1481 + return dict->pos < dict->limit;

1482 +}

1483 +

1484 +/*

1485 + * Get a byte from the dictionary at the given distance. The distance is

1486 + * assumed to valid, or as a special case, zero when the dictionary is

1487 + * still empty. This special case is needed for single-call decoding to

1488 + * avoid writing a '\0' to the end of the destination buffer.

1489 + */

1490 +static inline uint32_t dict_get(const struct dictionary *dict, uint32_t dist)

1491 +{

1492 + size_t offset = dict->pos - dist - 1;

1493 +

1494 + if (dist >= dict->pos)

1495 + offset += dict->end;

1496 +

1497 + return dict->full > 0 ? dict->buf[offset] : 0;

1498 +}

1499 +

1500 +/*

1501 + * Put one byte into the dictionary. It is assumed that there is space for it.

1502 + */

1503 +static inline void dict_put(struct dictionary *dict, uint8_t byte)

1504 +{

1505 + dict->buf[dict->pos++] = byte;

1506 +

1507 + if (dict->full < dict->pos)

1508 + dict->full = dict->pos;

1509 +}

1510 +

1511 +/*

1512 + * Repeat given number of bytes from the given distance. If the distance is

1513 + * invalid, false is returned. On success, true is returned and *len is

1514 + * updated to indicate how many bytes were left to be repeated.

1515 + */

1516 +static bool dict_repeat(struct dictionary *dict, uint32_t *len, uint32_t dist)

1517 +{

1518 + size_t back;

1519 + uint32_t left;

1520 +

1521 + if (dist >= dict->full || dist >= dict->size)

1522 + return false;

1523 +

1524 + left = min_t(size_t, dict->limit - dict->pos, *len);

1525 + *len -= left;

1526 +

1527 + back = dict->pos - dist - 1;

1528 + if (dist >= dict->pos)

1529 + back += dict->end;

1530 +

1531 + do {

1532 + dict->buf[dict->pos++] = dict->buf[back++];

1533 + if (back == dict->end)

1534 + back = 0;

1535 + } while (--left > 0);

1536 +

1537 + if (dict->full < dict->pos)

1538 + dict->full = dict->pos;

1539 +

1540 + return true;

1541 +}

1542 +

1543 +/* Copy uncompressed data as is from input to dictionary and output buffers. */

1544 +static void dict_uncompressed(struct dictionary *dict, struct xz_buf *b,

1545 + uint32_t *left)

1546 +{

1547 + size_t copy_size;

1548 +

1549 + while (*left > 0 && b->in_pos < b->in_size

1550 + && b->out_pos < b->out_size) {

1551 + copy_size = min(b->in_size - b->in_pos,

1552 + b->out_size - b->out_pos);

1553 + if (copy_size > dict->end - dict->pos)

1554 + copy_size = dict->end - dict->pos;

1555 + if (copy_size > *left)

1556 + copy_size = *left;

1557 +

1558 + *left -= copy_size;

1559 +

1560 + memcpy(dict->buf + dict->pos, b->in + b->in_pos, copy_size);

1561 + dict->pos += copy_size;

1562 +

1563 + if (dict->full < dict->pos)

1564 + dict->full = dict->pos;

1565 +

1566 + if (DEC_IS_MULTI(dict->mode)) {

1567 + if (dict->pos == dict->end)

1568 + dict->pos = 0;

1569 +

1570 + memcpy(b->out + b->out_pos, b->in + b->in_pos,

1571 + copy_size);

1572 + }

1573 +

1574 + dict->start = dict->pos;

1575 +

1576 + b->out_pos += copy_size;

1577 + b->in_pos += copy_size;

1578 + }

1579 +}

1580 +

1581 +/*

1582 + * Flush pending data from dictionary to b->out. It is assumed that there is

1583 + * enough space in b->out. This is guaranteed because caller uses dict_limit()

1584 + * before decoding data into the dictionary.

1585 + */

1586 +static uint32_t dict_flush(struct dictionary *dict, struct xz_buf *b)

1587 +{

1588 + size_t copy_size = dict->pos - dict->start;

1589 +

1590 + if (DEC_IS_MULTI(dict->mode)) {

1591 + if (dict->pos == dict->end)

1592 + dict->pos = 0;

1593 +

1594 + memcpy(b->out + b->out_pos, dict->buf + dict->start,

1595 + copy_size);

1596 + }

1597 +

1598 + dict->start = dict->pos;

1599 + b->out_pos += copy_size;

1600 + return copy_size;

1601 +}

1602 +

1603 +/*****************

1604 + * Range decoder *

1605 + *****************/

1606 +

1607 +/* Reset the range decoder. */

1608 +static void rc_reset(struct rc_dec *rc)

1609 +{

1610 + rc->range = (uint32_t)-1;

1611 + rc->code = 0;

1612 + rc->init_bytes_left = RC_INIT_BYTES;

1613 +}

1614 +

1615 +/*

1616 + * Read the first five initial bytes into rc->code if they haven't been

1617 + * read already. (Yes, the first byte gets completely ignored.)

1618 + */

1619 +static bool rc_read_init(struct rc_dec *rc, struct xz_buf *b)

1620 +{

1621 + while (rc->init_bytes_left > 0) {

1622 + if (b->in_pos == b->in_size)

1623 + return false;

1624 +

1625 + rc->code = (rc->code << 8) + b->in[b->in_pos++];

1626 + --rc->init_bytes_left;

1627 + }

1628 +

1629 + return true;

1630 +}

1631 +

1632 +/* Return true if there may not be enough input for the next decoding loop. */

1633 +static inline bool rc_limit_exceeded(const struct rc_dec *rc)

1634 +{

1635 + return rc->in_pos > rc->in_limit;

1636 +}

1637 +

1638 +/*

1639 + * Return true if it is possible (from point of view of range decoder) that

1640 + * we have reached the end of the LZMA chunk.

1641 + */

1642 +static inline bool rc_is_finished(const struct rc_dec *rc)

1643 +{

1644 + return rc->code == 0;

1645 +}

1646 +

1647 +/* Read the next input byte if needed. */

1648 +static __always_inline void rc_normalize(struct rc_dec *rc)

1649 +{

1650 + if (rc->range < RC_TOP_VALUE) {

1651 + rc->range <<= RC_SHIFT_BITS;

1652 + rc->code = (rc->code << RC_SHIFT_BITS) + rc->in[rc->in_pos++];

1653 + }

1654 +}

1655 +

1656 +/*

1657 + * Decode one bit. In some versions, this function has been splitted in three

1658 + * functions so that the compiler is supposed to be able to more easily avoid

1659 + * an extra branch. In this particular version of the LZMA decoder, this

1660 + * doesn't seem to be a good idea (tested with GCC 3.3.6, 3.4.6, and 4.3.3

1661 + * on x86). Using a non-splitted version results in nicer looking code too.

1662 + *

1663 + * NOTE: This must return an int. Do not make it return a bool or the speed

1664 + * of the code generated by GCC 3.x decreases 10-15 %. (GCC 4.3 doesn't care,

1665 + * and it generates 10-20 % faster code than GCC 3.x from this file anyway.)

1666 + */

1667 +static __always_inline int rc_bit(struct rc_dec *rc, uint16_t *prob)

1668 +{

1669 + uint32_t bound;

1670 + int bit;

1671 +

1672 + rc_normalize(rc);

1673 + bound = (rc->range >> RC_BIT_MODEL_TOTAL_BITS) * *prob;

1674 + if (rc->code < bound) {

1675 + rc->range = bound;

1676 + *prob += (RC_BIT_MODEL_TOTAL - *prob) >> RC_MOVE_BITS;

1677 + bit = 0;

1678 + } else {

1679 + rc->range -= bound;

1680 + rc->code -= bound;

1681 + *prob -= *prob >> RC_MOVE_BITS;

1682 + bit = 1;

1683 + }

1684 +

1685 + return bit;

1686 +}

1687 +

1688 +/* Decode a bittree starting from the most significant bit. */

1689 +static __always_inline uint32_t rc_bittree(struct rc_dec *rc,

1690 + uint16_t *probs, uint32_t limit)

1691 +{

1692 + uint32_t symbol = 1;

1693 +

1694 + do {

1695 + if (rc_bit(rc, &probs[symbol]))

1696 + symbol = (symbol << 1) + 1;

1697 + else

1698 + symbol <<= 1;

1699 + } while (symbol < limit);

1700 +

1701 + return symbol;

1702 +}

1703 +

1704 +/* Decode a bittree starting from the least significant bit. */

1705 +static __always_inline void rc_bittree_reverse(struct rc_dec *rc,

1706 + uint16_t *probs,

1707 + uint32_t *dest, uint32_t limit)

1708 +{

1709 + uint32_t symbol = 1;

1710 + uint32_t i = 0;

1711 +

1712 + do {

1713 + if (rc_bit(rc, &probs[symbol])) {

1714 + symbol = (symbol << 1) + 1;

1715 + *dest += 1 << i;

1716 + } else {

1717 + symbol <<= 1;

1718 + }

1719 + } while (++i < limit);

1720 +}

1721 +

1722 +/* Decode direct bits (fixed fifty-fifty probability) */

1723 +static inline void rc_direct(struct rc_dec *rc, uint32_t *dest, uint32_t limit)

1724 +{

1725 + uint32_t mask;

1726 +

1727 + do {

1728 + rc_normalize(rc);

1729 + rc->range >>= 1;

1730 + rc->code -= rc->range;

1731 + mask = (uint32_t)0 - (rc->code >> 31);

1732 + rc->code += rc->range & mask;

1733 + *dest = (*dest << 1) + (mask + 1);

1734 + } while (--limit > 0);

1735 +}

1736 +

1737 +/********

1738 + * LZMA *

1739 + ********/

1740 +

1741 +/* Get pointer to literal coder probability array. */

1742 +static uint16_t *lzma_literal_probs(struct xz_dec_lzma2 *s)

1743 +{

1744 + uint32_t prev_byte = dict_get(&s->dict, 0);

1745 + uint32_t low = prev_byte >> (8 - s->lzma.lc);

1746 + uint32_t high = (s->dict.pos & s->lzma.literal_pos_mask) << s->lzma.lc;

1747 + return s->lzma.literal[low + high];

1748 +}

1749 +

1750 +/* Decode a literal (one 8-bit byte) */

1751 +static void lzma_literal(struct xz_dec_lzma2 *s)

1752 +{

1753 + uint16_t *probs;

1754 + uint32_t symbol;

1755 + uint32_t match_byte;

1756 + uint32_t match_bit;

1757 + uint32_t offset;

1758 + uint32_t i;

1759 +

1760 + probs = lzma_literal_probs(s);

1761 +

1762 + if (lzma_state_is_literal(s->lzma.state)) {

1763 + symbol = rc_bittree(&s->rc, probs, 0x100);

1764 + } else {

1765 + symbol = 1;

1766 + match_byte = dict_get(&s->dict, s->lzma.rep0) << 1;

1767 + offset = 0x100;

1768 +

1769 + do {

1770 + match_bit = match_byte & offset;

1771 + match_byte <<= 1;

1772 + i = offset + match_bit + symbol;

1773 +

1774 + if (rc_bit(&s->rc, &probs[i])) {

1775 + symbol = (symbol << 1) + 1;

1776 + offset &= match_bit;

1777 + } else {

1778 + symbol <<= 1;

1779 + offset &= ~match_bit;

1780 + }

1781 + } while (symbol < 0x100);

1782 + }

1783 +

1784 + dict_put(&s->dict, (uint8_t)symbol);

1785 + lzma_state_literal(&s->lzma.state);

1786 +}

1787 +

1788 +/* Decode the length of the match into s->lzma.len. */

1789 +static void lzma_len(struct xz_dec_lzma2 *s, struct lzma_len_dec *l,

1790 + uint32_t pos_state)

1791 +{

1792 + uint16_t *probs;

1793 + uint32_t limit;

1794 +

1795 + if (!rc_bit(&s->rc, &l->choice)) {

1796 + probs = l->low[pos_state];

1797 + limit = LEN_LOW_SYMBOLS;

1798 + s->lzma.len = MATCH_LEN_MIN;

1799 + } else {

1800 + if (!rc_bit(&s->rc, &l->choice2)) {

1801 + probs = l->mid[pos_state];

1802 + limit = LEN_MID_SYMBOLS;

1803 + s->lzma.len = MATCH_LEN_MIN + LEN_LOW_SYMBOLS;

1804 + } else {

1805 + probs = l->high;

1806 + limit = LEN_HIGH_SYMBOLS;

1807 + s->lzma.len = MATCH_LEN_MIN + LEN_LOW_SYMBOLS

1808 + + LEN_MID_SYMBOLS;

1809 + }

1810 + }

1811 +

1812 + s->lzma.len += rc_bittree(&s->rc, probs, limit) - limit;

1813 +}

1814 +

1815 +/* Decode a match. The distance will be stored in s->lzma.rep0. */

1816 +static void lzma_match(struct xz_dec_lzma2 *s, uint32_t pos_state)

1817 +{

1818 + uint16_t *probs;

1819 + uint32_t dist_slot;

1820 + uint32_t limit;

1821 +

1822 + lzma_state_match(&s->lzma.state);

1823 +

1824 + s->lzma.rep3 = s->lzma.rep2;

1825 + s->lzma.rep2 = s->lzma.rep1;

1826 + s->lzma.rep1 = s->lzma.rep0;

1827 +

1828 + lzma_len(s, &s->lzma.match_len_dec, pos_state);

1829 +

1830 + probs = s->lzma.dist_slot[lzma_get_dist_state(s->lzma.len)];

1831 + dist_slot = rc_bittree(&s->rc, probs, DIST_SLOTS) - DIST_SLOTS;

1832 +

1833 + if (dist_slot < DIST_MODEL_START) {

1834 + s->lzma.rep0 = dist_slot;

1835 + } else {

1836 + limit = (dist_slot >> 1) - 1;

1837 + s->lzma.rep0 = 2 + (dist_slot & 1);

1838 +

1839 + if (dist_slot < DIST_MODEL_END) {

1840 + s->lzma.rep0 <<= limit;

1841 + probs = s->lzma.dist_special + s->lzma.rep0

1842 + - dist_slot - 1;

1843 + rc_bittree_reverse(&s->rc, probs,

1844 + &s->lzma.rep0, limit);

1845 + } else {

1846 + rc_direct(&s->rc, &s->lzma.rep0, limit - ALIGN_BITS);

1847 + s->lzma.rep0 <<= ALIGN_BITS;

1848 + rc_bittree_reverse(&s->rc, s->lzma.dist_align,

1849 + &s->lzma.rep0, ALIGN_BITS);

1850 + }

1851 + }

1852 +}

1853 +

1854 +/*

1855 + * Decode a repeated match. The distance is one of the four most recently

1856 + * seen matches. The distance will be stored in s->lzma.rep0.

1857 + */

1858 +static void lzma_rep_match(struct xz_dec_lzma2 *s, uint32_t pos_state)

1859 +{

1860 + uint32_t tmp;

1861 +

1862 + if (!rc_bit(&s->rc, &s->lzma.is_rep0[s->lzma.state])) {

1863 + if (!rc_bit(&s->rc, &s->lzma.is_rep0_long[

1864 + s->lzma.state][pos_state])) {

1865 + lzma_state_short_rep(&s->lzma.state);

1866 + s->lzma.len = 1;

1867 + return;

1868 + }

1869 + } else {

1870 + if (!rc_bit(&s->rc, &s->lzma.is_rep1[s->lzma.state])) {

1871 + tmp = s->lzma.rep1;

1872 + } else {

1873 + if (!rc_bit(&s->rc, &s->lzma.is_rep2[s->lzma.state])) {

1874 + tmp = s->lzma.rep2;

1875 + } else {

1876 + tmp = s->lzma.rep3;

1877 + s->lzma.rep3 = s->lzma.rep2;

1878 + }

1879 +

1880 + s->lzma.rep2 = s->lzma.rep1;

1881 + }

1882 +

1883 + s->lzma.rep1 = s->lzma.rep0;

1884 + s->lzma.rep0 = tmp;

1885 + }

1886 +

1887 + lzma_state_long_rep(&s->lzma.state);

1888 + lzma_len(s, &s->lzma.rep_len_dec, pos_state);

1889 +}

1890 +

1891 +/* LZMA decoder core */

1892 +static bool lzma_main(struct xz_dec_lzma2 *s)

1893 +{

1894 + uint32_t pos_state;

1895 +

1896 + /*

1897 + * If the dictionary was reached during the previous call, try to

1898 + * finish the possibly pending repeat in the dictionary.

1899 + */

1900 + if (dict_has_space(&s->dict) && s->lzma.len > 0)

1901 + dict_repeat(&s->dict, &s->lzma.len, s->lzma.rep0);

1902 +

1903 + /*

1904 + * Decode more LZMA symbols. One iteration may consume up to

1905 + * LZMA_IN_REQUIRED - 1 bytes.

1906 + */

1907 + while (dict_has_space(&s->dict) && !rc_limit_exceeded(&s->rc)) {

1908 + pos_state = s->dict.pos & s->lzma.pos_mask;

1909 +

1910 + if (!rc_bit(&s->rc, &s->lzma.is_match[

1911 + s->lzma.state][pos_state])) {

1912 + lzma_literal(s);

1913 + } else {

1914 + if (rc_bit(&s->rc, &s->lzma.is_rep[s->lzma.state]))

1915 + lzma_rep_match(s, pos_state);

1916 + else

1917 + lzma_match(s, pos_state);

1918 +

1919 + if (!dict_repeat(&s->dict, &s->lzma.len, s->lzma.rep0))

1920 + return false;

1921 + }

1922 + }

1923 +

1924 + /*

1925 + * Having the range decoder always normalized when we are outside

1926 + * this function makes it easier to correctly handle end of the chunk.

1927 + */

1928 + rc_normalize(&s->rc);

1929 +

1930 + return true;

1931 +}

1932 +

1933 +/*

1934 + * Reset the LZMA decoder and range decoder state. Dictionary is nore reset

1935 + * here, because LZMA state may be reset without resetting the dictionary.

1936 + */

1937 +static void lzma_reset(struct xz_dec_lzma2 *s)

1938 +{

1939 + uint16_t *probs;

1940 + size_t i;

1941 +

1942 + s->lzma.state = STATE_LIT_LIT;

1943 + s->lzma.rep0 = 0;

1944 + s->lzma.rep1 = 0;

1945 + s->lzma.rep2 = 0;

1946 + s->lzma.rep3 = 0;

1947 +

1948 + /*

1949 + * All probabilities are initialized to the same value. This hack

1950 + * makes the code smaller by avoiding a separate loop for each

1951 + * probability array.

1952 + *

1953 + * This could be optimized so that only that part of literal

1954 + * probabilities that are actually required. In the common case

1955 + * we would write 12 KiB less.

1956 + */

1957 + probs = s->lzma.is_match[0];

1958 + for (i = 0; i < PROBS_TOTAL; ++i)

1959 + probs[i] = RC_BIT_MODEL_TOTAL / 2;

1960 +

1961 + rc_reset(&s->rc);

1962 +}

1963 +

1964 +/*

1965 + * Decode and validate LZMA properties (lc/lp/pb) and calculate the bit masks

1966 + * from the decoded lp and pb values. On success, the LZMA decoder state is

1967 + * reset and true is returned.

1968 + */

1969 +static bool lzma_props(struct xz_dec_lzma2 *s, uint8_t props)

1970 +{

1971 + if (props > (4 * 5 + 4) * 9 + 8)

1972 + return false;

1973 +

1974 + s->lzma.pos_mask = 0;

1975 + while (props >= 9 * 5) {

1976 + props -= 9 * 5;

1977 + ++s->lzma.pos_mask;

1978 + }

1979 +

1980 + s->lzma.pos_mask = (1 << s->lzma.pos_mask) - 1;

1981 +

1982 + s->lzma.literal_pos_mask = 0;

1983 + while (props >= 9) {

1984 + props -= 9;

1985 + ++s->lzma.literal_pos_mask;

1986 + }

1987 +

1988 + s->lzma.lc = props;

1989 +

1990 + if (s->lzma.lc + s->lzma.literal_pos_mask > 4)

1991 + return false;

1992 +

1993 + s->lzma.literal_pos_mask = (1 << s->lzma.literal_pos_mask) - 1;

1994 +

1995 + lzma_reset(s);

1996 +

1997 + return true;

1998 +}

1999 +

2000 +/*********

2001 + * LZMA2 *

2002 + *********/

2003 +

2004 +/*

2005 + * The LZMA decoder assumes that if the input limit (s->rc.in_limit) hasn't

2006 + * been exceeded, it is safe to read up to LZMA_IN_REQUIRED bytes. This

2007 + * wrapper function takes care of making the LZMA decoder's assumption safe.

2008 + *

2009 + * As long as there is plenty of input left to be decoded in the current LZMA

2010 + * chunk, we decode directly from the caller-supplied input buffer until

2011 + * there's LZMA_IN_REQUIRED bytes left. Those remaining bytes are copied into

2012 + * s->temp.buf, which (hopefully) gets filled on the next call to this

2013 + * function. We decode a few bytes from the temporary buffer so that we can

2014 + * continue decoding from the caller-supplied input buffer again.

2015 + */

2016 +static bool lzma2_lzma(struct xz_dec_lzma2 *s, struct xz_buf *b)

2017 +{

2018 + size_t in_avail;

2019 + uint32_t tmp;

2020 +

2021 + in_avail = b->in_size - b->in_pos;

2022 + if (s->temp.size > 0 || s->lzma2.compressed == 0) {

2023 + tmp = 2 * LZMA_IN_REQUIRED - s->temp.size;

2024 + if (tmp > s->lzma2.compressed - s->temp.size)

2025 + tmp = s->lzma2.compressed - s->temp.size;

2026 + if (tmp > in_avail)

2027 + tmp = in_avail;

2028 +

2029 + memcpy(s->temp.buf + s->temp.size, b->in + b->in_pos, tmp);

2030 +

2031 + if (s->temp.size + tmp == s->lzma2.compressed) {

2032 + memzero(s->temp.buf + s->temp.size + tmp,

2033 + sizeof(s->temp.buf)

2034 + - s->temp.size - tmp);

2035 + s->rc.in_limit = s->temp.size + tmp;

2036 + } else if (s->temp.size + tmp < LZMA_IN_REQUIRED) {

2037 + s->temp.size += tmp;

2038 + b->in_pos += tmp;

2039 + return true;

2040 + } else {

2041 + s->rc.in_limit = s->temp.size + tmp - LZMA_IN_REQUIRED;

2042 + }

2043 +

2044 + s->rc.in = s->temp.buf;

2045 + s->rc.in_pos = 0;

2046 +

2047 + if (!lzma_main(s) || s->rc.in_pos > s->temp.size + tmp)

2048 + return false;

2049 +

2050 + s->lzma2.compressed -= s->rc.in_pos;

2051 +

2052 + if (s->rc.in_pos < s->temp.size) {

2053 + s->temp.size -= s->rc.in_pos;

2054 + memmove(s->temp.buf, s->temp.buf + s->rc.in_pos,

2055 + s->temp.size);

2056 + return true;

2057 + }

2058 +

2059 + b->in_pos += s->rc.in_pos - s->temp.size;

2060 + s->temp.size = 0;

2061 + }

2062 +

2063 + in_avail = b->in_size - b->in_pos;

2064 + if (in_avail >= LZMA_IN_REQUIRED) {

2065 + s->rc.in = b->in;

2066 + s->rc.in_pos = b->in_pos;

2067 +

2068 + if (in_avail >= s->lzma2.compressed + LZMA_IN_REQUIRED)

2069 + s->rc.in_limit = b->in_pos + s->lzma2.compressed;

2070 + else

2071 + s->rc.in_limit = b->in_size - LZMA_IN_REQUIRED;

2072 +

2073 + if (!lzma_main(s))

2074 + return false;

2075 +

2076 + in_avail = s->rc.in_pos - b->in_pos;

2077 + if (in_avail > s->lzma2.compressed)

2078 + return false;

2079 +

2080 + s->lzma2.compressed -= in_avail;

2081 + b->in_pos = s->rc.in_pos;

2082 + }

2083 +

2084 + in_avail = b->in_size - b->in_pos;

2085 + if (in_avail < LZMA_IN_REQUIRED) {

2086 + if (in_avail > s->lzma2.compressed)

2087 + in_avail = s->lzma2.compressed;

2088 +

2089 + memcpy(s->temp.buf, b->in + b->in_pos, in_avail);

2090 + s->temp.size = in_avail;

2091 + b->in_pos += in_avail;

2092 + }

2093 +

2094 + return true;

2095 +}

2096 +

2097 +/*

2098 + * Take care of the LZMA2 control layer, and forward the job of actual LZMA

2099 + * decoding or copying of uncompressed chunks to other functions.

2100 + */

2101 +XZ_EXTERN enum xz_ret xz_dec_lzma2_run(struct xz_dec_lzma2 *s,

2102 + struct xz_buf *b)

2103 +{

2104 + uint32_t tmp;

2105 +

2106 + while (b->in_pos < b->in_size || s->lzma2.sequence == SEQ_LZMA_RUN) {

2107 + switch (s->lzma2.sequence) {

2108 + case SEQ_CONTROL:

2109 + /*

2110 + * LZMA2 control byte

2111 + *

2112 + * Exact values:

2113 + * 0x00 End marker

2114 + * 0x01 Dictionary reset followed by

2115 + * an uncompressed chunk

2116 + * 0x02 Uncompressed chunk (no dictionary reset)

2117 + *

2118 + * Highest three bits (s->control & 0xE0):

2119 + * 0xE0 Dictionary reset, new properties and state

2120 + * reset, followed by LZMA compressed chunk

2121 + * 0xC0 New properties and state reset, followed

2122 + * by LZMA compressed chunk (no dictionary

2123 + * reset)

2124 + * 0xA0 State reset using old properties,

2125 + * followed by LZMA compressed chunk (no

2126 + * dictionary reset)

2127 + * 0x80 LZMA chunk (no dictionary or state reset)

2128 + *

2129 + * For LZMA compressed chunks, the lowest five bits

2130 + * (s->control & 1F) are the highest bits of the

2131 + * uncompressed size (bits 16-20).

2132 + *

2133 + * A new LZMA2 stream must begin with a dictionary

2134 + * reset. The first LZMA chunk must set new

2135 + * properties and reset the LZMA state.

2136 + *

2137 + * Values that don't match anything described above

2138 + * are invalid and we return XZ_DATA_ERROR.

2139 + */

2140 + tmp = b->in[b->in_pos++];

2141 +

2142 + if (tmp >= 0xE0 || tmp == 0x01) {

2143 + s->lzma2.need_props = true;

2144 + s->lzma2.need_dict_reset = false;

2145 + dict_reset(&s->dict, b);

2146 + } else if (s->lzma2.need_dict_reset) {

2147 + return XZ_DATA_ERROR;

2148 + }

2149 +

2150 + if (tmp >= 0x80) {

2151 + s->lzma2.uncompressed = (tmp & 0x1F) << 16;

2152 + s->lzma2.sequence = SEQ_UNCOMPRESSED_1;

2153 +

2154 + if (tmp >= 0xC0) {

2155 + /*

2156 + * When there are new properties,

2157 + * state reset is done at

2158 + * SEQ_PROPERTIES.

2159 + */

2160 + s->lzma2.need_props = false;

2161 + s->lzma2.next_sequence

2162 + = SEQ_PROPERTIES;

2163 +

2164 + } else if (s->lzma2.need_props) {

2165 + return XZ_DATA_ERROR;

2166 +

2167 + } else {

2168 + s->lzma2.next_sequence

2169 + = SEQ_LZMA_PREPARE;

2170 + if (tmp >= 0xA0)

2171 + lzma_reset(s);

2172 + }

2173 + } else {

2174 + if (tmp == 0x00)

2175 + return XZ_STREAM_END;

2176 +

2177 + if (tmp > 0x02)

2178 + return XZ_DATA_ERROR;

2179 +

2180 + s->lzma2.sequence = SEQ_COMPRESSED_0;

2181 + s->lzma2.next_sequence = SEQ_COPY;

2182 + }

2183 +

2184 + break;

2185 +

2186 + case SEQ_UNCOMPRESSED_1:

2187 + s->lzma2.uncompressed

2188 + += (uint32_t)b->in[b->in_pos++] << 8;

2189 + s->lzma2.sequence = SEQ_UNCOMPRESSED_2;

2190 + break;

2191 +

2192 + case SEQ_UNCOMPRESSED_2:

2193 + s->lzma2.uncompressed

2194 + += (uint32_t)b->in[b->in_pos++] + 1;

2195 + s->lzma2.sequence = SEQ_COMPRESSED_0;

2196 + break;

2197 +

2198 + case SEQ_COMPRESSED_0:

2199 + s->lzma2.compressed

2200 + = (uint32_t)b->in[b->in_pos++] << 8;

2201 + s->lzma2.sequence = SEQ_COMPRESSED_1;

2202 + break;

2203 +

2204 + case SEQ_COMPRESSED_1:

2205 + s->lzma2.compressed

2206 + += (uint32_t)b->in[b->in_pos++] + 1;

2207 + s->lzma2.sequence = s->lzma2.next_sequence;

2208 + break;

2209 +

2210 + case SEQ_PROPERTIES:

2211 + if (!lzma_props(s, b->in[b->in_pos++]))

2212 + return XZ_DATA_ERROR;

2213 +

2214 + s->lzma2.sequence = SEQ_LZMA_PREPARE;

2215 +

2216 + case SEQ_LZMA_PREPARE:

2217 + if (s->lzma2.compressed < RC_INIT_BYTES)

2218 + return XZ_DATA_ERROR;

2219 +

2220 + if (!rc_read_init(&s->rc, b))

2221 + return XZ_OK;

2222 +

2223 + s->lzma2.compressed -= RC_INIT_BYTES;

2224 + s->lzma2.sequence = SEQ_LZMA_RUN;

2225 +

2226 + case SEQ_LZMA_RUN:

2227 + /*

2228 + * Set dictionary limit to indicate how much we want

2229 + * to be encoded at maximum. Decode new data into the

2230 + * dictionary. Flush the new data from dictionary to

2231 + * b->out. Check if we finished decoding this chunk.

2232 + * In case the dictionary got full but we didn't fill

2233 + * the output buffer yet, we may run this loop

2234 + * multiple times without changing s->lzma2.sequence.

2235 + */

2236 + dict_limit(&s->dict, min_t(size_t,

2237 + b->out_size - b->out_pos,

2238 + s->lzma2.uncompressed));

2239 + if (!lzma2_lzma(s, b))

2240 + return XZ_DATA_ERROR;

2241 +

2242 + s->lzma2.uncompressed -= dict_flush(&s->dict, b);

2243 +

2244 + if (s->lzma2.uncompressed == 0) {

2245 + if (s->lzma2.compressed > 0 || s->lzma.len > 0

2246 + || !rc_is_finished(&s->rc))

2247 + return XZ_DATA_ERROR;

2248 +

2249 + rc_reset(&s->rc);

2250 + s->lzma2.sequence = SEQ_CONTROL;

2251 +

2252 + } else if (b->out_pos == b->out_size

2253 + || (b->in_pos == b->in_size

2254 + && s->temp.size

2255 + < s->lzma2.compressed)) {

2256 + return XZ_OK;

2257 + }

2258 +

2259 + break;

2260 +

2261 + case SEQ_COPY:

2262 + dict_uncompressed(&s->dict, b, &s->lzma2.compressed);

2263 + if (s->lzma2.compressed > 0)

2264 + return XZ_OK;

2265 +

2266 + s->lzma2.sequence = SEQ_CONTROL;

2267 + break;

2268 + }

2269 + }

2270 +

2271 + return XZ_OK;

2272 +}

2273 +

2274 +XZ_EXTERN struct xz_dec_lzma2 *xz_dec_lzma2_create(enum xz_mode mode,

2275 + uint32_t dict_max)

2276 +{

2277 + struct xz_dec_lzma2 *s = kmalloc(sizeof(*s), GFP_KERNEL);

2278 + if (s == NULL)

2279 + return NULL;

2280 +

2281 + s->dict.mode = mode;

2282 + s->dict.size_max = dict_max;

2283 +

2284 + if (DEC_IS_PREALLOC(mode)) {

2285 + s->dict.buf = vmalloc(dict_max);

2286 + if (s->dict.buf == NULL) {

2287 + kfree(s);

2288 + return NULL;

2289 + }

2290 + } else if (DEC_IS_DYNALLOC(mode)) {

2291 + s->dict.buf = NULL;

2292 + s->dict.allocated = 0;

2293 + }

2294 +

2295 + return s;

2296 +}

2297 +

2298 +XZ_EXTERN enum xz_ret xz_dec_lzma2_reset(struct xz_dec_lzma2 *s, uint8_t props)

2299 +{

2300 + /* This limits dictionary size to 3 GiB to keep parsing simpler. */

2301 + if (props > 39)

2302 + return XZ_OPTIONS_ERROR;

2303 +

2304 + s->dict.size = 2 + (props & 1);

2305 + s->dict.size <<= (props >> 1) + 11;

2306 +

2307 + if (DEC_IS_MULTI(s->dict.mode)) {

2308 + if (s->dict.size > s->dict.size_max)

2309 + return XZ_MEMLIMIT_ERROR;

2310 +

2311 + s->dict.end = s->dict.size;

2312 +

2313 + if (DEC_IS_DYNALLOC(s->dict.mode)) {

2314 + if (s->dict.allocated < s->dict.size) {

2315 + vfree(s->dict.buf);

2316 + s->dict.buf = vmalloc(s->dict.size);

2317 + if (s->dict.buf == NULL) {

2318 + s->dict.allocated = 0;

2319 + return XZ_MEM_ERROR;

2320 + }

2321 + }

2322 + }

2323 + }

2324 +

2325 + s->lzma.len = 0;

2326 +

2327 + s->lzma2.sequence = SEQ_CONTROL;

2328 + s->lzma2.need_dict_reset = true;

2329 +

2330 + s->temp.size = 0;

2331 +

2332 + return XZ_OK;

2333 +}

2334 +

2335 +XZ_EXTERN void xz_dec_lzma2_end(struct xz_dec_lzma2 *s)

2336 +{

2337 + if (DEC_IS_MULTI(s->dict.mode))

2338 + vfree(s->dict.buf);

2339 +

2340 + kfree(s);

2341 +}

2342 diff --git a/lib/xz/xz_dec_stream.c b/lib/xz/xz_dec_stream.c

2343 new file mode 100644

2344 index 0000000..ac809b1

2345 --- /dev/null

2346 +++ b/lib/xz/xz_dec_stream.c

2347 @@ -0,0 +1,821 @@

2348 +/*

2349 + * .xz Stream decoder

2350 + *

2351 + * Author: Lasse Collin <lasse.collin@tukaani.org>

2352 + *

2353 + * This file has been put into the public domain.

2354 + * You can do whatever you want with this file.

2355 + */

2356 +

2357 +#include "xz_private.h"

2358 +#include "xz_stream.h"

2359 +

2360 +/* Hash used to validate the Index field */

2361 +struct xz_dec_hash {

2362 + vli_type unpadded;

2363 + vli_type uncompressed;

2364 + uint32_t crc32;

2365 +};

2366 +

2367 +struct xz_dec {

2368 + /* Position in dec_main() */

2369 + enum {

2370 + SEQ_STREAM_HEADER,

2371 + SEQ_BLOCK_START,

2372 + SEQ_BLOCK_HEADER,

2373 + SEQ_BLOCK_UNCOMPRESS,

2374 + SEQ_BLOCK_PADDING,

2375 + SEQ_BLOCK_CHECK,

2376 + SEQ_INDEX,

2377 + SEQ_INDEX_PADDING,

2378 + SEQ_INDEX_CRC32,

2379 + SEQ_STREAM_FOOTER

2380 + } sequence;

2381 +

2382 + /* Position in variable-length integers and Check fields */

2383 + uint32_t pos;

2384 +

2385 + /* Variable-length integer decoded by dec_vli() */

2386 + vli_type vli;

2387 +

2388 + /* Saved in_pos and out_pos */

2389 + size_t in_start;

2390 + size_t out_start;

2391 +

2392 + /* CRC32 value in Block or Index */

2393 + uint32_t crc32;

2394 +

2395 + /* Type of the integrity check calculated from uncompressed data */

2396 + enum xz_check check_type;

2397 +

2398 + /* Operation mode */

2399 + enum xz_mode mode;

2400 +

2401 + /*

2402 + * True if the next call to xz_dec_run() is allowed to return

2403 + * XZ_BUF_ERROR.

2404 + */

2405 + bool allow_buf_error;

2406 +

2407 + /* Information stored in Block Header */

2408 + struct {

2409 + /*

2410 + * Value stored in the Compressed Size field, or

2411 + * VLI_UNKNOWN if Compressed Size is not present.

2412 + */

2413 + vli_type compressed;

2414 +

2415 + /*

2416 + * Value stored in the Uncompressed Size field, or

2417 + * VLI_UNKNOWN if Uncompressed Size is not present.

2418 + */

2419 + vli_type uncompressed;

2420 +

2421 + /* Size of the Block Header field */

2422 + uint32_t size;

2423 + } block_header;

2424 +

2425 + /* Information collected when decoding Blocks */

2426 + struct {

2427 + /* Observed compressed size of the current Block */

2428 + vli_type compressed;

2429 +

2430 + /* Observed uncompressed size of the current Block */

2431 + vli_type uncompressed;

2432 +

2433 + /* Number of Blocks decoded so far */

2434 + vli_type count;

2435 +

2436 + /*

2437 + * Hash calculated from the Block sizes. This is used to

2438 + * validate the Index field.

2439 + */

2440 + struct xz_dec_hash hash;

2441 + } block;

2442 +

2443 + /* Variables needed when verifying the Index field */

2444 + struct {

2445 + /* Position in dec_index() */

2446 + enum {

2447 + SEQ_INDEX_COUNT,

2448 + SEQ_INDEX_UNPADDED,

2449 + SEQ_INDEX_UNCOMPRESSED

2450 + } sequence;

2451 +

2452 + /* Size of the Index in bytes */

2453 + vli_type size;

2454 +

2455 + /* Number of Records (matches block.count in valid files) */

2456 + vli_type count;

2457 +

2458 + /*

2459 + * Hash calculated from the Records (matches block.hash in

2460 + * valid files).

2461 + */

2462 + struct xz_dec_hash hash;

2463 + } index;

2464 +

2465 + /*

2466 + * Temporary buffer needed to hold Stream Header, Block Header,

2467 + * and Stream Footer. The Block Header is the biggest (1 KiB)

2468 + * so we reserve space according to that. buf[] has to be aligned

2469 + * to a multiple of four bytes; the size_t variables before it

2470 + * should guarantee this.

2471 + */

2472 + struct {

2473 + size_t pos;

2474 + size_t size;

2475 + uint8_t buf[1024];

2476 + } temp;

2477 +

2478 + struct xz_dec_lzma2 *lzma2;

2479 +

2480 +#ifdef XZ_DEC_BCJ

2481 + struct xz_dec_bcj *bcj;

2482 + bool bcj_active;

2483 +#endif

2484 +};

2485 +

2486 +#ifdef XZ_DEC_ANY_CHECK

2487 +/* Sizes of the Check field with different Check IDs */

2488 +static const uint8_t check_sizes[16] = {

2489 + 0,

2490 + 4, 4, 4,

2491 + 8, 8, 8,

2492 + 16, 16, 16,

2493 + 32, 32, 32,

2494 + 64, 64, 64

2495 +};

2496 +#endif

2497 +

2498 +/*

2499 + * Fill s->temp by copying data starting from b->in[b->in_pos]. Caller

2500 + * must have set s->temp.pos to indicate how much data we are supposed

2501 + * to copy into s->temp.buf. Return true once s->temp.pos has reached

2502 + * s->temp.size.

2503 + */

2504 +static bool fill_temp(struct xz_dec *s, struct xz_buf *b)

2505 +{

2506 + size_t copy_size = min_t(size_t,

2507 + b->in_size - b->in_pos, s->temp.size - s->temp.pos);

2508 +

2509 + memcpy(s->temp.buf + s->temp.pos, b->in + b->in_pos, copy_size);

2510 + b->in_pos += copy_size;

2511 + s->temp.pos += copy_size;

2512 +

2513 + if (s->temp.pos == s->temp.size) {

2514 + s->temp.pos = 0;

2515 + return true;

2516 + }

2517 +

2518 + return false;

2519 +}

2520 +

2521 +/* Decode a variable-length integer (little-endian base-128 encoding) */

2522 +static enum xz_ret dec_vli(struct xz_dec *s, const uint8_t *in,

2523 + size_t *in_pos, size_t in_size)

2524 +{

2525 + uint8_t byte;

2526 +

2527 + if (s->pos == 0)

2528 + s->vli = 0;

2529 +

2530 + while (*in_pos < in_size) {

2531 + byte = in[*in_pos];

2532 + ++*in_pos;

2533 +

2534 + s->vli |= (vli_type)(byte & 0x7F) << s->pos;

2535 +

2536 + if ((byte & 0x80) == 0) {

2537 + /* Don't allow non-minimal encodings. */

2538 + if (byte == 0 && s->pos != 0)

2539 + return XZ_DATA_ERROR;

2540 +

2541 + s->pos = 0;

2542 + return XZ_STREAM_END;

2543 + }

2544 +

2545 + s->pos += 7;

2546 + if (s->pos == 7 * VLI_BYTES_MAX)

2547 + return XZ_DATA_ERROR;

2548 + }

2549 +

2550 + return XZ_OK;

2551 +}

2552 +

2553 +/*

2554 + * Decode the Compressed Data field from a Block. Update and validate

2555 + * the observed compressed and uncompressed sizes of the Block so that

2556 + * they don't exceed the values possibly stored in the Block Header

2557 + * (validation assumes that no integer overflow occurs, since vli_type

2558 + * is normally uint64_t). Update the CRC32 if presence of the CRC32

2559 + * field was indicated in Stream Header.

2560 + *

2561 + * Once the decoding is finished, validate that the observed sizes match

2562 + * the sizes possibly stored in the Block Header. Update the hash and

2563 + * Block count, which are later used to validate the Index field.

2564 + */

2565 +static enum xz_ret dec_block(struct xz_dec *s, struct xz_buf *b)

2566 +{

2567 + enum xz_ret ret;

2568 +

2569 + s->in_start = b->in_pos;

2570 + s->out_start = b->out_pos;

2571 +

2572 +#ifdef XZ_DEC_BCJ

2573 + if (s->bcj_active)

2574 + ret = xz_dec_bcj_run(s->bcj, s->lzma2, b);

2575 + else

2576 +#endif

2577 + ret = xz_dec_lzma2_run(s->lzma2, b);

2578 +

2579 + s->block.compressed += b->in_pos - s->in_start;

2580 + s->block.uncompressed += b->out_pos - s->out_start;

2581 +

2582 + /*

2583 + * There is no need to separately check for VLI_UNKNOWN, since

2584 + * the observed sizes are always smaller than VLI_UNKNOWN.

2585 + */

2586 + if (s->block.compressed > s->block_header.compressed

2587 + || s->block.uncompressed

2588 + > s->block_header.uncompressed)

2589 + return XZ_DATA_ERROR;

2590 +

2591 + if (s->check_type == XZ_CHECK_CRC32)

2592 + s->crc32 = xz_crc32(b->out + s->out_start,

2593 + b->out_pos - s->out_start, s->crc32);

2594 +

2595 + if (ret == XZ_STREAM_END) {

2596 + if (s->block_header.compressed != VLI_UNKNOWN

2597 + && s->block_header.compressed

2598 + != s->block.compressed)

2599 + return XZ_DATA_ERROR;

2600 +

2601 + if (s->block_header.uncompressed != VLI_UNKNOWN

2602 + && s->block_header.uncompressed

2603 + != s->block.uncompressed)

2604 + return XZ_DATA_ERROR;

2605 +

2606 + s->block.hash.unpadded += s->block_header.size

2607 + + s->block.compressed;

2608 +

2609 +#ifdef XZ_DEC_ANY_CHECK

2610 + s->block.hash.unpadded += check_sizes[s->check_type];

2611 +#else

2612 + if (s->check_type == XZ_CHECK_CRC32)

2613 + s->block.hash.unpadded += 4;

2614 +#endif

2615 +

2616 + s->block.hash.uncompressed += s->block.uncompressed;

2617 + s->block.hash.crc32 = xz_crc32(

2618 + (const uint8_t *)&s->block.hash,

2619 + sizeof(s->block.hash), s->block.hash.crc32);

2620 +

2621 + ++s->block.count;

2622 + }

2623 +

2624 + return ret;

2625 +}

2626 +

2627 +/* Update the Index size and the CRC32 value. */

2628 +static void index_update(struct xz_dec *s, const struct xz_buf *b)

2629 +{

2630 + size_t in_used = b->in_pos - s->in_start;

2631 + s->index.size += in_used;

2632 + s->crc32 = xz_crc32(b->in + s->in_start, in_used, s->crc32);

2633 +}

2634 +

2635 +/*

2636 + * Decode the Number of Records, Unpadded Size, and Uncompressed Size

2637 + * fields from the Index field. That is, Index Padding and CRC32 are not

2638 + * decoded by this function.

2639 + *

2640 + * This can return XZ_OK (more input needed), XZ_STREAM_END (everything

2641 + * successfully decoded), or XZ_DATA_ERROR (input is corrupt).

2642 + */

2643 +static enum xz_ret dec_index(struct xz_dec *s, struct xz_buf *b)

2644 +{

2645 + enum xz_ret ret;

2646 +

2647 + do {

2648 + ret = dec_vli(s, b->in, &b->in_pos, b->in_size);

2649 + if (ret != XZ_STREAM_END) {

2650 + index_update(s, b);

2651 + return ret;

2652 + }

2653 +

2654 + switch (s->index.sequence) {

2655 + case SEQ_INDEX_COUNT:

2656 + s->index.count = s->vli;

2657 +

2658 + /*

2659 + * Validate that the Number of Records field

2660 + * indicates the same number of Records as

2661 + * there were Blocks in the Stream.

2662 + */

2663 + if (s->index.count != s->block.count)

2664 + return XZ_DATA_ERROR;

2665 +

2666 + s->index.sequence = SEQ_INDEX_UNPADDED;

2667 + break;

2668 +

2669 + case SEQ_INDEX_UNPADDED:

2670 + s->index.hash.unpadded += s->vli;

2671 + s->index.sequence = SEQ_INDEX_UNCOMPRESSED;

2672 + break;

2673 +

2674 + case SEQ_INDEX_UNCOMPRESSED:

2675 + s->index.hash.uncompressed += s->vli;

2676 + s->index.hash.crc32 = xz_crc32(

2677 + (const uint8_t *)&s->index.hash,

2678 + sizeof(s->index.hash),

2679 + s->index.hash.crc32);

2680 + --s->index.count;

2681 + s->index.sequence = SEQ_INDEX_UNPADDED;

2682 + break;

2683 + }

2684 + } while (s->index.count > 0);

2685 +

2686 + return XZ_STREAM_END;

2687 +}

2688 +

2689 +/*

2690 + * Validate that the next four input bytes match the value of s->crc32.

2691 + * s->pos must be zero when starting to validate the first byte.

2692 + */

2693 +static enum xz_ret crc32_validate(struct xz_dec *s, struct xz_buf *b)

2694 +{

2695 + do {

2696 + if (b->in_pos == b->in_size)

2697 + return XZ_OK;

2698 +

2699 + if (((s->crc32 >> s->pos) & 0xFF) != b->in[b->in_pos++])

2700 + return XZ_DATA_ERROR;

2701 +

2702 + s->pos += 8;

2703 +

2704 + } while (s->pos < 32);

2705 +

2706 + s->crc32 = 0;

2707 + s->pos = 0;

2708 +

2709 + return XZ_STREAM_END;

2710 +}

2711 +

2712 +#ifdef XZ_DEC_ANY_CHECK

2713 +/*

2714 + * Skip over the Check field when the Check ID is not supported.

2715 + * Returns true once the whole Check field has been skipped over.

2716 + */

2717 +static bool check_skip(struct xz_dec *s, struct xz_buf *b)

2718 +{

2719 + while (s->pos < check_sizes[s->check_type]) {

2720 + if (b->in_pos == b->in_size)

2721 + return false;

2722 +

2723 + ++b->in_pos;

2724 + ++s->pos;

2725 + }

2726 +

2727 + s->pos = 0;

2728 +

2729 + return true;

2730 +}

2731 +#endif

2732 +

2733 +/* Decode the Stream Header field (the first 12 bytes of the .xz Stream). */

2734 +static enum xz_ret dec_stream_header(struct xz_dec *s)

2735 +{

2736 + if (!memeq(s->temp.buf, HEADER_MAGIC, HEADER_MAGIC_SIZE))

2737 + return XZ_FORMAT_ERROR;

2738 +

2739 + if (xz_crc32(s->temp.buf + HEADER_MAGIC_SIZE, 2, 0)

2740 + != get_le32(s->temp.buf + HEADER_MAGIC_SIZE + 2))

2741 + return XZ_DATA_ERROR;

2742 +

2743 + if (s->temp.buf[HEADER_MAGIC_SIZE] != 0)

2744 + return XZ_OPTIONS_ERROR;

2745 +

2746 + /*

2747 + * Of integrity checks, we support only none (Check ID = 0) and

2748 + * CRC32 (Check ID = 1). However, if XZ_DEC_ANY_CHECK is defined,

2749 + * we will accept other check types too, but then the check won't

2750 + * be verified and a warning (XZ_UNSUPPORTED_CHECK) will be given.

2751 + */

2752 + s->check_type = s->temp.buf[HEADER_MAGIC_SIZE + 1];

2753 +

2754 +#ifdef XZ_DEC_ANY_CHECK

2755 + if (s->check_type > XZ_CHECK_MAX)

2756 + return XZ_OPTIONS_ERROR;

2757 +

2758 + if (s->check_type > XZ_CHECK_CRC32)

2759 + return XZ_UNSUPPORTED_CHECK;

2760 +#else

2761 + if (s->check_type > XZ_CHECK_CRC32)

2762 + return XZ_OPTIONS_ERROR;

2763 +#endif

2764 +

2765 + return XZ_OK;

2766 +}

2767 +

2768 +/* Decode the Stream Footer field (the last 12 bytes of the .xz Stream) */

2769 +static enum xz_ret dec_stream_footer(struct xz_dec *s)

2770 +{

2771 + if (!memeq(s->temp.buf + 10, FOOTER_MAGIC, FOOTER_MAGIC_SIZE))

2772 + return XZ_DATA_ERROR;

2773 +

2774 + if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf))

2775 + return XZ_DATA_ERROR;

2776 +

2777 + /*

2778 + * Validate Backward Size. Note that we never added the size of the

2779 + * Index CRC32 field to s->index.size, thus we use s->index.size / 4

2780 + * instead of s->index.size / 4 - 1.

2781 + */

2782 + if ((s->index.size >> 2) != get_le32(s->temp.buf + 4))

2783 + return XZ_DATA_ERROR;

2784 +

2785 + if (s->temp.buf[8] != 0 || s->temp.buf[9] != s->check_type)

2786 + return XZ_DATA_ERROR;

2787 +

2788 + /*

2789 + * Use XZ_STREAM_END instead of XZ_OK to be more convenient

2790 + * for the caller.

2791 + */

2792 + return XZ_STREAM_END;

2793 +}

2794 +

2795 +/* Decode the Block Header and initialize the filter chain. */

2796 +static enum xz_ret dec_block_header(struct xz_dec *s)

2797 +{

2798 + enum xz_ret ret;

2799 +

2800 + /*

2801 + * Validate the CRC32. We know that the temp buffer is at least

2802 + * eight bytes so this is safe.

2803 + */

2804 + s->temp.size -= 4;

2805 + if (xz_crc32(s->temp.buf, s->temp.size, 0)

2806 + != get_le32(s->temp.buf + s->temp.size))

2807 + return XZ_DATA_ERROR;

2808 +

2809 + s->temp.pos = 2;

2810 +

2811 + /*

2812 + * Catch unsupported Block Flags. We support only one or two filters

2813 + * in the chain, so we catch that with the same test.

2814 + */

2815 +#ifdef XZ_DEC_BCJ

2816 + if (s->temp.buf[1] & 0x3E)

2817 +#else

2818 + if (s->temp.buf[1] & 0x3F)

2819 +#endif

2820 + return XZ_OPTIONS_ERROR;

2821 +

2822 + /* Compressed Size */

2823 + if (s->temp.buf[1] & 0x40) {

2824 + if (dec_vli(s, s->temp.buf, &s->temp.pos, s->temp.size)

2825 + != XZ_STREAM_END)

2826 + return XZ_DATA_ERROR;

2827 +

2828 + s->block_header.compressed = s->vli;

2829 + } else {

2830 + s->block_header.compressed = VLI_UNKNOWN;

2831 + }

2832 +

2833 + /* Uncompressed Size */

2834 + if (s->temp.buf[1] & 0x80) {

2835 + if (dec_vli(s, s->temp.buf, &s->temp.pos, s->temp.size)

2836 + != XZ_STREAM_END)

2837 + return XZ_DATA_ERROR;

2838 +

2839 + s->block_header.uncompressed = s->vli;

2840 + } else {

2841 + s->block_header.uncompressed = VLI_UNKNOWN;

2842 + }

2843 +

2844 +#ifdef XZ_DEC_BCJ

2845 + /* If there are two filters, the first one must be a BCJ filter. */

2846 + s->bcj_active = s->temp.buf[1] & 0x01;

2847 + if (s->bcj_active) {

2848 + if (s->temp.size - s->temp.pos < 2)

2849 + return XZ_OPTIONS_ERROR;

2850 +

2851 + ret = xz_dec_bcj_reset(s->bcj, s->temp.buf[s->temp.pos++]);

2852 + if (ret != XZ_OK)

2853 + return ret;

2854 +

2855 + /*

2856 + * We don't support custom start offset,

2857 + * so Size of Properties must be zero.

2858 + */

2859 + if (s->temp.buf[s->temp.pos++] != 0x00)

2860 + return XZ_OPTIONS_ERROR;

2861 + }

2862 +#endif

2863 +

2864 + /* Valid Filter Flags always take at least two bytes. */

2865 + if (s->temp.size - s->temp.pos < 2)

2866 + return XZ_DATA_ERROR;

2867 +

2868 + /* Filter ID = LZMA2 */

2869 + if (s->temp.buf[s->temp.pos++] != 0x21)

2870 + return XZ_OPTIONS_ERROR;

2871 +

2872 + /* Size of Properties = 1-byte Filter Properties */

2873 + if (s->temp.buf[s->temp.pos++] != 0x01)

2874 + return XZ_OPTIONS_ERROR;

2875 +

2876 + /* Filter Properties contains LZMA2 dictionary size. */

2877 + if (s->temp.size - s->temp.pos < 1)

2878 + return XZ_DATA_ERROR;

2879 +

2880 + ret = xz_dec_lzma2_reset(s->lzma2, s->temp.buf[s->temp.pos++]);

2881 + if (ret != XZ_OK)

2882 + return ret;

2883 +

2884 + /* The rest must be Header Padding. */

2885 + while (s->temp.pos < s->temp.size)

2886 + if (s->temp.buf[s->temp.pos++] != 0x00)

2887 + return XZ_OPTIONS_ERROR;

2888 +

2889 + s->temp.pos = 0;

2890 + s->block.compressed = 0;

2891 + s->block.uncompressed = 0;

2892 +

2893 + return XZ_OK;

2894 +}

2895 +

2896 +static enum xz_ret dec_main(struct xz_dec *s, struct xz_buf *b)

2897 +{

2898 + enum xz_ret ret;

2899 +

2900 + /*

2901 + * Store the start position for the case when we are in the middle

2902 + * of the Index field.

2903 + */

2904 + s->in_start = b->in_pos;

2905 +

2906 + while (true) {

2907 + switch (s->sequence) {

2908 + case SEQ_STREAM_HEADER:

2909 + /*

2910 + * Stream Header is copied to s->temp, and then

2911 + * decoded from there. This way if the caller

2912 + * gives us only little input at a time, we can

2913 + * still keep the Stream Header decoding code

2914 + * simple. Similar approach is used in many places

2915 + * in this file.

2916 + */

2917 + if (!fill_temp(s, b))

2918 + return XZ_OK;

2919 +

2920 + /*

2921 + * If dec_stream_header() returns

2922 + * XZ_UNSUPPORTED_CHECK, it is still possible

2923 + * to continue decoding if working in multi-call

2924 + * mode. Thus, update s->sequence before calling

2925 + * dec_stream_header().

2926 + */

2927 + s->sequence = SEQ_BLOCK_START;

2928 +

2929 + ret = dec_stream_header(s);

2930 + if (ret != XZ_OK)

2931 + return ret;

2932 +

2933 + case SEQ_BLOCK_START:

2934 + /* We need one byte of input to continue. */

2935 + if (b->in_pos == b->in_size)

2936 + return XZ_OK;

2937 +

2938 + /* See if this is the beginning of the Index field. */

2939 + if (b->in[b->in_pos] == 0) {

2940 + s->in_start = b->in_pos++;

2941 + s->sequence = SEQ_INDEX;

2942 + break;

2943 + }

2944 +

2945 + /*

2946 + * Calculate the size of the Block Header and

2947 + * prepare to decode it.

2948 + */

2949 + s->block_header.size

2950 + = ((uint32_t)b->in[b->in_pos] + 1) * 4;

2951 +

2952 + s->temp.size = s->block_header.size;

2953 + s->temp.pos = 0;

2954 + s->sequence = SEQ_BLOCK_HEADER;

2955 +

2956 + case SEQ_BLOCK_HEADER:

2957 + if (!fill_temp(s, b))

2958 + return XZ_OK;

2959 +

2960 + ret = dec_block_header(s);

2961 + if (ret != XZ_OK)

2962 + return ret;

2963 +

2964 + s->sequence = SEQ_BLOCK_UNCOMPRESS;

2965 +

2966 + case SEQ_BLOCK_UNCOMPRESS:

2967 + ret = dec_block(s, b);

2968 + if (ret != XZ_STREAM_END)

2969 + return ret;

2970 +

2971 + s->sequence = SEQ_BLOCK_PADDING;

2972 +

2973 + case SEQ_BLOCK_PADDING:

2974 + /*

2975 + * Size of Compressed Data + Block Padding

2976 + * must be a multiple of four. We don't need

2977 + * s->block.compressed for anything else

2978 + * anymore, so we use it here to test the size

2979 + * of the Block Padding field.

2980 + */

2981 + while (s->block.compressed & 3) {

2982 + if (b->in_pos == b->in_size)

2983 + return XZ_OK;

2984 +

2985 + if (b->in[b->in_pos++] != 0)

2986 + return XZ_DATA_ERROR;

2987 +

2988 + ++s->block.compressed;

2989 + }

2990 +

2991 + s->sequence = SEQ_BLOCK_CHECK;

2992 +

2993 + case SEQ_BLOCK_CHECK:

2994 + if (s->check_type == XZ_CHECK_CRC32) {

2995 + ret = crc32_validate(s, b);

2996 + if (ret != XZ_STREAM_END)

2997 + return ret;

2998 + }

2999 +#ifdef XZ_DEC_ANY_CHECK

3000 + else if (!check_skip(s, b)) {

3001 + return XZ_OK;

3002 + }

3003 +#endif

3004 +

3005 + s->sequence = SEQ_BLOCK_START;

3006 + break;

3007 +

3008 + case SEQ_INDEX:

3009 + ret = dec_index(s, b);

3010 + if (ret != XZ_STREAM_END)

3011 + return ret;

3012 +

3013 + s->sequence = SEQ_INDEX_PADDING;

3014 +

3015 + case SEQ_INDEX_PADDING:

3016 + while ((s->index.size + (b->in_pos - s->in_start))

3017 + & 3) {

3018 + if (b->in_pos == b->in_size) {

3019 + index_update(s, b);

3020 + return XZ_OK;

3021 + }

3022 +

3023 + if (b->in[b->in_pos++] != 0)

3024 + return XZ_DATA_ERROR;

3025 + }

3026 +

3027 + /* Finish the CRC32 value and Index size. */

3028 + index_update(s, b);

3029 +

3030 + /* Compare the hashes to validate the Index field. */

3031 + if (!memeq(&s->block.hash, &s->index.hash,

3032 + sizeof(s->block.hash)))

3033 + return XZ_DATA_ERROR;

3034 +

3035 + s->sequence = SEQ_INDEX_CRC32;

3036 +

3037 + case SEQ_INDEX_CRC32:

3038 + ret = crc32_validate(s, b);

3039 + if (ret != XZ_STREAM_END)

3040 + return ret;

3041 +

3042 + s->temp.size = STREAM_HEADER_SIZE;

3043 + s->sequence = SEQ_STREAM_FOOTER;

3044 +

3045 + case SEQ_STREAM_FOOTER:

3046 + if (!fill_temp(s, b))

3047 + return XZ_OK;

3048 +

3049 + return dec_stream_footer(s);

3050 + }

3051 + }

3052 +

3053 + /* Never reached */

3054 +}

3055 +

3056 +/*

3057 + * xz_dec_run() is a wrapper for dec_main() to handle some special cases in

3058 + * multi-call and single-call decoding.

3059 + *

3060 + * In multi-call mode, we must return XZ_BUF_ERROR when it seems clear that we

3061 + * are not going to make any progress anymore. This is to prevent the caller

3062 + * from calling us infinitely when the input file is truncated or otherwise

3063 + * corrupt. Since zlib-style API allows that the caller fills the input buffer

3064 + * only when the decoder doesn't produce any new output, we have to be careful

3065 + * to avoid returning XZ_BUF_ERROR too easily: XZ_BUF_ERROR is returned only

3066 + * after the second consecutive call to xz_dec_run() that makes no progress.

3067 + *

3068 + * In single-call mode, if we couldn't decode everything and no error

3069 + * occurred, either the input is truncated or the output buffer is too small.

3070 + * Since we know that the last input byte never produces any output, we know

3071 + * that if all the input was consumed and decoding wasn't finished, the file

3072 + * must be corrupt. Otherwise the output buffer has to be too small or the

3073 + * file is corrupt in a way that decoding it produces too big output.

3074 + *

3075 + * If single-call decoding fails, we reset b->in_pos and b->out_pos back to

3076 + * their original values. This is because with some filter chains there won't

3077 + * be any valid uncompressed data in the output buffer unless the decoding

3078 + * actually succeeds (that's the price to pay of using the output buffer as

3079 + * the workspace).

3080 + */

3081 +XZ_EXTERN enum xz_ret xz_dec_run(struct xz_dec *s, struct xz_buf *b)

3082 +{

3083 + size_t in_start;

3084 + size_t out_start;

3085 + enum xz_ret ret;

3086 +

3087 + if (DEC_IS_SINGLE(s->mode))

3088 + xz_dec_reset(s);

3089 +

3090 + in_start = b->in_pos;

3091 + out_start = b->out_pos;

3092 + ret = dec_main(s, b);

3093 +

3094 + if (DEC_IS_SINGLE(s->mode)) {

3095 + if (ret == XZ_OK)

3096 + ret = b->in_pos == b->in_size

3097 + ? XZ_DATA_ERROR : XZ_BUF_ERROR;

3098 +

3099 + if (ret != XZ_STREAM_END) {

3100 + b->in_pos = in_start;

3101 + b->out_pos = out_start;

3102 + }

3103 +

3104 + } else if (ret == XZ_OK && in_start == b->in_pos

3105 + && out_start == b->out_pos) {

3106 + if (s->allow_buf_error)

3107 + ret = XZ_BUF_ERROR;

3108 +

3109 + s->allow_buf_error = true;

3110 + } else {

3111 + s->allow_buf_error = false;

3112 + }

3113 +

3114 + return ret;

3115 +}

3116 +

3117 +XZ_EXTERN struct xz_dec *xz_dec_init(enum xz_mode mode, uint32_t dict_max)

3118 +{

3119 + struct xz_dec *s = kmalloc(sizeof(*s), GFP_KERNEL);

3120 + if (s == NULL)

3121 + return NULL;

3122 +

3123 + s->mode = mode;

3124 +

3125 +#ifdef XZ_DEC_BCJ

3126 + s->bcj = xz_dec_bcj_create(DEC_IS_SINGLE(mode));

3127 + if (s->bcj == NULL)

3128 + goto error_bcj;

3129 +#endif

3130 +

3131 + s->lzma2 = xz_dec_lzma2_create(mode, dict_max);

3132 + if (s->lzma2 == NULL)

3133 + goto error_lzma2;

3134 +

3135 + xz_dec_reset(s);

3136 + return s;

3137 +

3138 +error_lzma2:

3139 +#ifdef XZ_DEC_BCJ

3140 + xz_dec_bcj_end(s->bcj);

3141 +error_bcj:

3142 +#endif

3143 + kfree(s);

3144 + return NULL;

3145 +}

3146 +

3147 +XZ_EXTERN void xz_dec_reset(struct xz_dec *s)

3148 +{

3149 + s->sequence = SEQ_STREAM_HEADER;

3150 + s->allow_buf_error = false;

3151 + s->pos = 0;

3152 + s->crc32 = 0;

3153 + memzero(&s->block, sizeof(s->block));

3154 + memzero(&s->index, sizeof(s->index));

3155 + s->temp.pos = 0;

3156 + s->temp.size = STREAM_HEADER_SIZE;

3157 +}

3158 +

3159 +XZ_EXTERN void xz_dec_end(struct xz_dec *s)

3160 +{

3161 + if (s != NULL) {

3162 + xz_dec_lzma2_end(s->lzma2);

3163 +#ifdef XZ_DEC_BCJ

3164 + xz_dec_bcj_end(s->bcj);

3165 +#endif

3166 + kfree(s);

3167 + }

3168 +}

3169 diff --git a/lib/xz/xz_dec_syms.c b/lib/xz/xz_dec_syms.c

3170 new file mode 100644

3171 index 0000000..32eb3c0

3172 --- /dev/null

3173 +++ b/lib/xz/xz_dec_syms.c

3174 @@ -0,0 +1,26 @@

3175 +/*

3176 + * XZ decoder module information

3177 + *

3178 + * Author: Lasse Collin <lasse.collin@tukaani.org>

3179 + *

3180 + * This file has been put into the public domain.

3181 + * You can do whatever you want with this file.

3182 + */

3183 +

3184 +#include <linux/module.h>

3185 +#include <linux/xz.h>

3186 +

3187 +EXPORT_SYMBOL(xz_dec_init);

3188 +EXPORT_SYMBOL(xz_dec_reset);

3189 +EXPORT_SYMBOL(xz_dec_run);

3190 +EXPORT_SYMBOL(xz_dec_end);

3191 +

3192 +MODULE_DESCRIPTION("XZ decompressor");

3193 +MODULE_VERSION("1.0");

3194 +MODULE_AUTHOR("Lasse Collin <lasse.collin@tukaani.org> and Igor Pavlov");

3195 +

3196 +/*

3197 + * This code is in the public domain, but in Linux it's simplest to just

3198 + * say it's GPL and consider the authors as the copyright holders.

3199 + */

3200 +MODULE_LICENSE("GPL");

3201 diff --git a/lib/xz/xz_dec_test.c b/lib/xz/xz_dec_test.c

3202 new file mode 100644

3203 index 0000000..da28a19

3204 --- /dev/null

3205 +++ b/lib/xz/xz_dec_test.c

3206 @@ -0,0 +1,220 @@

3207 +/*

3208 + * XZ decoder tester

3209 + *

3210 + * Author: Lasse Collin <lasse.collin@tukaani.org>

3211 + *

3212 + * This file has been put into the public domain.

3213 + * You can do whatever you want with this file.

3214 + */

3215 +

3216 +#include <linux/kernel.h>

3217 +#include <linux/module.h>

3218 +#include <linux/fs.h>

3219 +#include <linux/uaccess.h>

3220 +#include <linux/crc32.h>

3221 +#include <linux/xz.h>

3222 +

3223 +/* Maximum supported dictionary size */

3224 +#define DICT_MAX (1 << 20)

3225 +

3226 +/* Device name to pass to register_chrdev(). */

3227 +#define DEVICE_NAME "xz_dec_test"

3228 +

3229 +/* Dynamically allocated device major number */

3230 +static int device_major;

3231 +

3232 +/*

3233 + * We reuse the same decoder state, and thus can decode only one

3234 + * file at a time.

3235 + */

3236 +static bool device_is_open;

3237 +

3238 +/* XZ decoder state */

3239 +static struct xz_dec *state;

3240 +

3241 +/*

3242 + * Return value of xz_dec_run(). We need to avoid calling xz_dec_run() after

3243 + * it has returned XZ_STREAM_END, so we make this static.

3244 + */

3245 +static enum xz_ret ret;

3246 +

3247 +/*

3248 + * Input and output buffers. The input buffer is used as a temporary safe

3249 + * place for the data coming from the userspace.

3250 + */

3251 +static uint8_t buffer_in[1024];

3252 +static uint8_t buffer_out[1024];

3253 +

3254 +/*

3255 + * Structure to pass the input and output buffers to the XZ decoder.

3256 + * A few of the fields are never modified so we initialize them here.

3257 + */

3258 +static struct xz_buf buffers = {

3259 + .in = buffer_in,

3260 + .out = buffer_out,

3261 + .out_size = sizeof(buffer_out)

3262 +};

3263 +

3264 +/*

3265 + * CRC32 of uncompressed data. This is used to give the user a simple way

3266 + * to check that the decoder produces correct output.

3267 + */

3268 +static uint32_t crc;

3269 +

3270 +static int xz_dec_test_open(struct inode *i, struct file *f)

3271 +{

3272 + if (device_is_open)

3273 + return -EBUSY;

3274 +

3275 + device_is_open = true;

3276 +

3277 + xz_dec_reset(state);

3278 + ret = XZ_OK;

3279 + crc = 0xFFFFFFFF;

3280 +

3281 + buffers.in_pos = 0;

3282 + buffers.in_size = 0;

3283 + buffers.out_pos = 0;

3284 +

3285 + printk(KERN_INFO DEVICE_NAME ": opened\n");

3286 + return 0;

3287 +}

3288 +

3289 +static int xz_dec_test_release(struct inode *i, struct file *f)

3290 +{

3291 + device_is_open = false;

3292 +

3293 + if (ret == XZ_OK)

3294 + printk(KERN_INFO DEVICE_NAME ": input was truncated\n");

3295 +

3296 + printk(KERN_INFO DEVICE_NAME ": closed\n");

3297 + return 0;

3298 +}

3299 +

3300 +/*

3301 + * Decode the data given to us from the userspace. CRC32 of the uncompressed

3302 + * data is calculated and is printed at the end of successful decoding. The

3303 + * uncompressed data isn't stored anywhere for further use.

3304 + *

3305 + * The .xz file must have exactly one Stream and no Stream Padding. The data

3306 + * after the first Stream is considered to be garbage.

3307 + */

3308 +static ssize_t xz_dec_test_write(struct file *file, const char __user *buf,

3309 + size_t size, loff_t *pos)

3310 +{

3311 + size_t remaining;

3312 +

3313 + if (ret != XZ_OK) {

3314 + if (size > 0)

3315 + printk(KERN_INFO DEVICE_NAME ": %zu bytes of "

3316 + "garbage at the end of the file\n",

3317 + size);

3318 +

3319 + return -ENOSPC;

3320 + }

3321 +

3322 + printk(KERN_INFO DEVICE_NAME ": decoding %zu bytes of input\n",

3323 + size);

3324 +

3325 + remaining = size;

3326 + while ((remaining > 0 || buffers.out_pos == buffers.out_size)

3327 + && ret == XZ_OK) {

3328 + if (buffers.in_pos == buffers.in_size) {

3329 + buffers.in_pos = 0;

3330 + buffers.in_size = min(remaining, sizeof(buffer_in));

3331 + if (copy_from_user(buffer_in, buf, buffers.in_size))

3332 + return -EFAULT;

3333 +

3334 + buf += buffers.in_size;

3335 + remaining -= buffers.in_size;

3336 + }

3337 +

3338 + buffers.out_pos = 0;

3339 + ret = xz_dec_run(state, &buffers);

3340 + crc = crc32(crc, buffer_out, buffers.out_pos);

3341 + }

3342 +

3343 + switch (ret) {

3344 + case XZ_OK:

3345 + printk(KERN_INFO DEVICE_NAME ": XZ_OK\n");

3346 + return size;

3347 +

3348 + case XZ_STREAM_END:

3349 + printk(KERN_INFO DEVICE_NAME ": XZ_STREAM_END, "

3350 + "CRC32 = 0x%08X\n", ~crc);

3351 + return size - remaining - (buffers.in_size - buffers.in_pos);

3352 +

3353 + case XZ_MEMLIMIT_ERROR:

3354 + printk(KERN_INFO DEVICE_NAME ": XZ_MEMLIMIT_ERROR\n");

3355 + break;

3356 +

3357 + case XZ_FORMAT_ERROR:

3358 + printk(KERN_INFO DEVICE_NAME ": XZ_FORMAT_ERROR\n");

3359 + break;

3360 +

3361 + case XZ_OPTIONS_ERROR:

3362 + printk(KERN_INFO DEVICE_NAME ": XZ_OPTIONS_ERROR\n");

3363 + break;

3364 +

3365 + case XZ_DATA_ERROR:

3366 + printk(KERN_INFO DEVICE_NAME ": XZ_DATA_ERROR\n");

3367 + break;

3368 +

3369 + case XZ_BUF_ERROR:

3370 + printk(KERN_INFO DEVICE_NAME ": XZ_BUF_ERROR\n");

3371 + break;

3372 +

3373 + default:

3374 + printk(KERN_INFO DEVICE_NAME ": Bug detected!\n");

3375 + break;

3376 + }

3377 +

3378 + return -EIO;

3379 +}

3380 +

3381 +/* Allocate the XZ decoder state and register the character device. */

3382 +static int __init xz_dec_test_init(void)

3383 +{

3384 + static const struct file_operations fileops = {

3385 + .owner = THIS_MODULE,

3386 + .open = &xz_dec_test_open,

3387 + .release = &xz_dec_test_release,

3388 + .write = &xz_dec_test_write

3389 + };

3390 +

3391 + state = xz_dec_init(XZ_PREALLOC, DICT_MAX);

3392 + if (state == NULL)

3393 + return -ENOMEM;

3394 +

3395 + device_major = register_chrdev(0, DEVICE_NAME, &fileops);

3396 + if (device_major < 0) {

3397 + xz_dec_end(state);

3398 + return device_major;

3399 + }

3400 +

3401 + printk(KERN_INFO DEVICE_NAME ": module loaded\n");

3402 + printk(KERN_INFO DEVICE_NAME ": Create a device node with "

3403 + "'mknod " DEVICE_NAME " c %d 0' and write .xz files "

3404 + "to it.\n", device_major);

3405 + return 0;

3406 +}

3407 +

3408 +static void __exit xz_dec_test_exit(void)

3409 +{

3410 + unregister_chrdev(device_major, DEVICE_NAME);

3411 + xz_dec_end(state);

3412 + printk(KERN_INFO DEVICE_NAME ": module unloaded\n");

3413 +}

3414 +

3415 +module_init(xz_dec_test_init);

3416 +module_exit(xz_dec_test_exit);

3417 +

3418 +MODULE_DESCRIPTION("XZ decompressor tester");

3419 +MODULE_VERSION("1.0");

3420 +MODULE_AUTHOR("Lasse Collin <lasse.collin@tukaani.org>");

3421 +

3422 +/*

3423 + * This code is in the public domain, but in Linux it's simplest to just

3424 + * say it's GPL and consider the authors as the copyright holders.

3425 + */

3426 +MODULE_LICENSE("GPL");

3427 diff --git a/lib/xz/xz_lzma2.h b/lib/xz/xz_lzma2.h

3428 new file mode 100644

3429 index 0000000..071d67b

3430 --- /dev/null

3431 +++ b/lib/xz/xz_lzma2.h

3432 @@ -0,0 +1,204 @@

3433 +/*

3434 + * LZMA2 definitions

3435 + *

3436 + * Authors: Lasse Collin <lasse.collin@tukaani.org>

3437 + * Igor Pavlov <http://7-zip.org/>

3438 + *

3439 + * This file has been put into the public domain.

3440 + * You can do whatever you want with this file.

3441 + */

3442 +

3443 +#ifndef XZ_LZMA2_H

3444 +#define XZ_LZMA2_H

3445 +

3446 +/* Range coder constants */

3447 +#define RC_SHIFT_BITS 8

3448 +#define RC_TOP_BITS 24

3449 +#define RC_TOP_VALUE (1 << RC_TOP_BITS)

3450 +#define RC_BIT_MODEL_TOTAL_BITS 11

3451 +#define RC_BIT_MODEL_TOTAL (1 << RC_BIT_MODEL_TOTAL_BITS)

3452 +#define RC_MOVE_BITS 5

3453 +

3454 +/*

3455 + * Maximum number of position states. A position state is the lowest pb

3456 + * number of bits of the current uncompressed offset. In some places there

3457 + * are different sets of probabilities for different position states.

3458 + */

3459 +#define POS_STATES_MAX (1 << 4)

3460 +

3461 +/*

3462 + * This enum is used to track which LZMA symbols have occurred most recently

3463 + * and in which order. This information is used to predict the next symbol.

3464 + *

3465 + * Symbols:

3466 + * - Literal: One 8-bit byte

3467 + * - Match: Repeat a chunk of data at some distance

3468 + * - Long repeat: Multi-byte match at a recently seen distance

3469 + * - Short repeat: One-byte repeat at a recently seen distance

3470 + *

3471 + * The symbol names are in from STATE_oldest_older_previous. REP means

3472 + * either short or long repeated match, and NONLIT means any non-literal.

3473 + */

3474 +enum lzma_state {

3475 + STATE_LIT_LIT,

3476 + STATE_MATCH_LIT_LIT,

3477 + STATE_REP_LIT_LIT,

3478 + STATE_SHORTREP_LIT_LIT,

3479 + STATE_MATCH_LIT,

3480 + STATE_REP_LIT,

3481 + STATE_SHORTREP_LIT,

3482 + STATE_LIT_MATCH,

3483 + STATE_LIT_LONGREP,

3484 + STATE_LIT_SHORTREP,

3485 + STATE_NONLIT_MATCH,

3486 + STATE_NONLIT_REP

3487 +};

3488 +

3489 +/* Total number of states */

3490 +#define STATES 12

3491 +

3492 +/* The lowest 7 states indicate that the previous state was a literal. */

3493 +#define LIT_STATES 7

3494 +

3495 +/* Indicate that the latest symbol was a literal. */

3496 +static inline void lzma_state_literal(enum lzma_state *state)

3497 +{

3498 + if (*state <= STATE_SHORTREP_LIT_LIT)

3499 + *state = STATE_LIT_LIT;

3500 + else if (*state <= STATE_LIT_SHORTREP)

3501 + *state -= 3;

3502 + else

3503 + *state -= 6;

3504 +}

3505 +

3506 +/* Indicate that the latest symbol was a match. */

3507 +static inline void lzma_state_match(enum lzma_state *state)

3508 +{

3509 + *state = *state < LIT_STATES ? STATE_LIT_MATCH : STATE_NONLIT_MATCH;

3510 +}

3511 +

3512 +/* Indicate that the latest state was a long repeated match. */

3513 +static inline void lzma_state_long_rep(enum lzma_state *state)

3514 +{

3515 + *state = *state < LIT_STATES ? STATE_LIT_LONGREP : STATE_NONLIT_REP;

3516 +}

3517 +

3518 +/* Indicate that the latest symbol was a short match. */

3519 +static inline void lzma_state_short_rep(enum lzma_state *state)

3520 +{

3521 + *state = *state < LIT_STATES ? STATE_LIT_SHORTREP : STATE_NONLIT_REP;

3522 +}

3523 +

3524 +/* Test if the previous symbol was a literal. */

3525 +static inline bool lzma_state_is_literal(enum lzma_state state)

3526 +{

3527 + return state < LIT_STATES;

3528 +}

3529 +

3530 +/* Each literal coder is divided in three sections:

3531 + * - 0x001-0x0FF: Without match byte

3532 + * - 0x101-0x1FF: With match byte; match bit is 0

3533 + * - 0x201-0x2FF: With match byte; match bit is 1

3534 + *

3535 + * Match byte is used when the previous LZMA symbol was something else than

3536 + * a literal (that is, it was some kind of match).

3537 + */

3538 +#define LITERAL_CODER_SIZE 0x300

3539 +

3540 +/* Maximum number of literal coders */

3541 +#define LITERAL_CODERS_MAX (1 << 4)

3542 +

3543 +/* Minimum length of a match is two bytes. */

3544 +#define MATCH_LEN_MIN 2

3545 +

3546 +/* Match length is encoded with 4, 5, or 10 bits.

3547 + *

3548 + * Length Bits

3549 + * 2-9 4 = Choice=0 + 3 bits

3550 + * 10-17 5 = Choice=1 + Choice2=0 + 3 bits

3551 + * 18-273 10 = Choice=1 + Choice2=1 + 8 bits

3552 + */

3553 +#define LEN_LOW_BITS 3

3554 +#define LEN_LOW_SYMBOLS (1 << LEN_LOW_BITS)

3555 +#define LEN_MID_BITS 3

3556 +#define LEN_MID_SYMBOLS (1 << LEN_MID_BITS)

3557 +#define LEN_HIGH_BITS 8

3558 +#define LEN_HIGH_SYMBOLS (1 << LEN_HIGH_BITS)

3559 +#define LEN_SYMBOLS (LEN_LOW_SYMBOLS + LEN_MID_SYMBOLS + LEN_HIGH_SYMBOLS)

3560 +

3561 +/*

3562 + * Maximum length of a match is 273 which is a result of the encoding

3563 + * described above.

3564 + */

3565 +#define MATCH_LEN_MAX (MATCH_LEN_MIN + LEN_SYMBOLS - 1)

3566 +

3567 +/*

3568 + * Different sets of probabilities are used for match distances that have

3569 + * very short match length: Lengths of 2, 3, and 4 bytes have a separate

3570 + * set of probabilities for each length. The matches with longer length

3571 + * use a shared set of probabilities.

3572 + */

3573 +#define DIST_STATES 4

3574 +

3575 +/*

3576 + * Get the index of the appropriate probability array for decoding

3577 + * the distance slot.

3578 + */

3579 +static inline uint32_t lzma_get_dist_state(uint32_t len)

3580 +{

3581 + return len < DIST_STATES + MATCH_LEN_MIN

3582 + ? len - MATCH_LEN_MIN : DIST_STATES - 1;

3583 +}

3584 +

3585 +/*

3586 + * The highest two bits of a 32-bit match distance are encoded using six bits.

3587 + * This six-bit value is called a distance slot. This way encoding a 32-bit

3588 + * value takes 6-36 bits, larger values taking more bits.

3589 + */

3590 +#define DIST_SLOT_BITS 6

3591 +#define DIST_SLOTS (1 << DIST_SLOT_BITS)

3592 +

3593 +/* Match distances up to 127 are fully encoded using probabilities. Since

3594 + * the highest two bits (distance slot) are always encoded using six bits,

3595 + * the distances 0-3 don't need any additional bits to encode, since the

3596 + * distance slot itself is the same as the actual distance. DIST_MODEL_START

3597 + * indicates the first distance slot where at least one additional bit is

3598 + * needed.

3599 + */

3600 +#define DIST_MODEL_START 4

3601 +

3602 +/*

3603 + * Match distances greater than 127 are encoded in three pieces:

3604 + * - distance slot: the highest two bits

3605 + * - direct bits: 2-26 bits below the highest two bits

3606 + * - alignment bits: four lowest bits

3607 + *

3608 + * Direct bits don't use any probabilities.

3609 + *

3610 + * The distance slot value of 14 is for distances 128-191.

3611 + */

3612 +#define DIST_MODEL_END 14

3613 +

3614 +/* Distance slots that indicate a distance <= 127. */

3615 +#define FULL_DISTANCES_BITS (DIST_MODEL_END / 2)

3616 +#define FULL_DISTANCES (1 << FULL_DISTANCES_BITS)

3617 +

3618 +/*

3619 + * For match distances greater than 127, only the highest two bits and the

3620 + * lowest four bits (alignment) is encoded using probabilities.

3621 + */

3622 +#define ALIGN_BITS 4

3623 +#define ALIGN_SIZE (1 << ALIGN_BITS)

3624 +#define ALIGN_MASK (ALIGN_SIZE - 1)

3625 +

3626 +/* Total number of all probability variables */

3627 +#define PROBS_TOTAL (1846 + LITERAL_CODERS_MAX * LITERAL_CODER_SIZE)

3628 +

3629 +/*

3630 + * LZMA remembers the four most recent match distances. Reusing these

3631 + * distances tends to take less space than re-encoding the actual

3632 + * distance value.

3633 + */

3634 +#define REPS 4

3635 +

3636 +#endif

3637 diff --git a/lib/xz/xz_private.h b/lib/xz/xz_private.h

3638 new file mode 100644

3639 index 0000000..a65633e

3640 --- /dev/null

3641 +++ b/lib/xz/xz_private.h

3642 @@ -0,0 +1,156 @@

3643 +/*

3644 + * Private includes and definitions

3645 + *

3646 + * Author: Lasse Collin <lasse.collin@tukaani.org>

3647 + *

3648 + * This file has been put into the public domain.

3649 + * You can do whatever you want with this file.

3650 + */

3651 +

3652 +#ifndef XZ_PRIVATE_H

3653 +#define XZ_PRIVATE_H

3654 +

3655 +#ifdef __KERNEL__

3656 +# include <linux/xz.h>

3657 +# include <asm/byteorder.h>

3658 +# include <asm/unaligned.h>

3659 + /* XZ_PREBOOT may be defined only via decompress_unxz.c. */

3660 +# ifndef XZ_PREBOOT

3661 +# include <linux/slab.h>

3662 +# include <linux/vmalloc.h>

3663 +# include <linux/string.h>

3664 +# ifdef CONFIG_XZ_DEC_X86

3665 +# define XZ_DEC_X86

3666 +# endif

3667 +# ifdef CONFIG_XZ_DEC_POWERPC

3668 +# define XZ_DEC_POWERPC

3669 +# endif

3670 +# ifdef CONFIG_XZ_DEC_IA64

3671 +# define XZ_DEC_IA64

3672 +# endif

3673 +# ifdef CONFIG_XZ_DEC_ARM

3674 +# define XZ_DEC_ARM

3675 +# endif

3676 +# ifdef CONFIG_XZ_DEC_ARMTHUMB

3677 +# define XZ_DEC_ARMTHUMB

3678 +# endif

3679 +# ifdef CONFIG_XZ_DEC_SPARC

3680 +# define XZ_DEC_SPARC

3681 +# endif

3682 +# define memeq(a, b, size) (memcmp(a, b, size) == 0)

3683 +# define memzero(buf, size) memset(buf, 0, size)

3684 +# endif

3685 +# define get_le32(p) le32_to_cpup((const uint32_t *)(p))

3686 +#else

3687 + /*

3688 + * For userspace builds, use a separate header to define the required

3689 + * macros and functions. This makes it easier to adapt the code into

3690 + * different environments and avoids clutter in the Linux kernel tree.

3691 + */

3692 +# include "xz_config.h"

3693 +#endif

3694 +

3695 +/* If no specific decoding mode is requested, enable support for all modes. */

3696 +#if !defined(XZ_DEC_SINGLE) && !defined(XZ_DEC_PREALLOC) \

3697 + && !defined(XZ_DEC_DYNALLOC)

3698 +# define XZ_DEC_SINGLE

3699 +# define XZ_DEC_PREALLOC

3700 +# define XZ_DEC_DYNALLOC

3701 +#endif

3702 +

3703 +/*

3704 + * The DEC_IS_foo(mode) macros are used in "if" statements. If only some

3705 + * of the supported modes are enabled, these macros will evaluate to true or

3706 + * false at compile time and thus allow the compiler to omit unneeded code.

3707 + */

3708 +#ifdef XZ_DEC_SINGLE

3709 +# define DEC_IS_SINGLE(mode) ((mode) == XZ_SINGLE)

3710 +#else

3711 +# define DEC_IS_SINGLE(mode) (false)

3712 +#endif

3713 +

3714 +#ifdef XZ_DEC_PREALLOC

3715 +# define DEC_IS_PREALLOC(mode) ((mode) == XZ_PREALLOC)

3716 +#else

3717 +# define DEC_IS_PREALLOC(mode) (false)

3718 +#endif

3719 +

3720 +#ifdef XZ_DEC_DYNALLOC

3721 +# define DEC_IS_DYNALLOC(mode) ((mode) == XZ_DYNALLOC)

3722 +#else

3723 +# define DEC_IS_DYNALLOC(mode) (false)

3724 +#endif

3725 +

3726 +#if !defined(XZ_DEC_SINGLE)

3727 +# define DEC_IS_MULTI(mode) (true)

3728 +#elif defined(XZ_DEC_PREALLOC) || defined(XZ_DEC_DYNALLOC)

3729 +# define DEC_IS_MULTI(mode) ((mode) != XZ_SINGLE)

3730 +#else

3731 +# define DEC_IS_MULTI(mode) (false)

3732 +#endif

3733 +

3734 +/*

3735 + * If any of the BCJ filter decoders are wanted, define XZ_DEC_BCJ.

3736 + * XZ_DEC_BCJ is used to enable generic support for BCJ decoders.

3737 + */

3738 +#ifndef XZ_DEC_BCJ

3739 +# if defined(XZ_DEC_X86) || defined(XZ_DEC_POWERPC) \

3740 + || defined(XZ_DEC_IA64) || defined(XZ_DEC_ARM) \

3741 + || defined(XZ_DEC_ARM) || defined(XZ_DEC_ARMTHUMB) \

3742 + || defined(XZ_DEC_SPARC)

3743 +# define XZ_DEC_BCJ

3744 +# endif

3745 +#endif

3746 +

3747 +/*

3748 + * Allocate memory for LZMA2 decoder. xz_dec_lzma2_reset() must be used

3749 + * before calling xz_dec_lzma2_run().

3750 + */

3751 +XZ_EXTERN struct xz_dec_lzma2 *xz_dec_lzma2_create(enum xz_mode mode,

3752 + uint32_t dict_max);

3753 +

3754 +/*

3755 + * Decode the LZMA2 properties (one byte) and reset the decoder. Return

3756 + * XZ_OK on success, XZ_MEMLIMIT_ERROR if the preallocated dictionary is not

3757 + * big enough, and XZ_OPTIONS_ERROR if props indicates something that this

3758 + * decoder doesn't support.

3759 + */

3760 +XZ_EXTERN enum xz_ret xz_dec_lzma2_reset(struct xz_dec_lzma2 *s,

3761 + uint8_t props);

3762 +

3763 +/* Decode raw LZMA2 stream from b->in to b->out. */

3764 +XZ_EXTERN enum xz_ret xz_dec_lzma2_run(struct xz_dec_lzma2 *s,

3765 + struct xz_buf *b);

3766 +

3767 +/* Free the memory allocated for the LZMA2 decoder. */

3768 +XZ_EXTERN void xz_dec_lzma2_end(struct xz_dec_lzma2 *s);

3769 +

3770 +#ifdef XZ_DEC_BCJ

3771 +/*

3772 + * Allocate memory for BCJ decoders. xz_dec_bcj_reset() must be used before

3773 + * calling xz_dec_bcj_run().

3774 + */

3775 +XZ_EXTERN struct xz_dec_bcj *xz_dec_bcj_create(bool single_call);

3776 +

3777 +/*

3778 + * Decode the Filter ID of a BCJ filter. This implementation doesn't

3779 + * support custom start offsets, so no decoding of Filter Properties

3780 + * is needed. Returns XZ_OK if the given Filter ID is supported.

3781 + * Otherwise XZ_OPTIONS_ERROR is returned.

3782 + */

3783 +XZ_EXTERN enum xz_ret xz_dec_bcj_reset(struct xz_dec_bcj *s, uint8_t id);

3784 +

3785 +/*

3786 + * Decode raw BCJ + LZMA2 stream. This must be used only if there actually is

3787 + * a BCJ filter in the chain. If the chain has only LZMA2, xz_dec_lzma2_run()

3788 + * must be called directly.

3789 + */

3790 +XZ_EXTERN enum xz_ret xz_dec_bcj_run(struct xz_dec_bcj *s,

3791 + struct xz_dec_lzma2 *lzma2,

3792 + struct xz_buf *b);

3793 +

3794 +/* Free the memory allocated for the BCJ filters. */

3795 +#define xz_dec_bcj_end(s) kfree(s)

3796 +#endif

3797 +

3798 +#endif

3799 diff --git a/lib/xz/xz_stream.h b/lib/xz/xz_stream.h

3800 new file mode 100644

3801 index 0000000..66cb5a7

3802 --- /dev/null

3803 +++ b/lib/xz/xz_stream.h

3804 @@ -0,0 +1,62 @@

3805 +/*

3806 + * Definitions for handling the .xz file format

3807 + *

3808 + * Author: Lasse Collin <lasse.collin@tukaani.org>

3809 + *

3810 + * This file has been put into the public domain.

3811 + * You can do whatever you want with this file.

3812 + */

3813 +

3814 +#ifndef XZ_STREAM_H

3815 +#define XZ_STREAM_H

3816 +

3817 +#if defined(__KERNEL__) && !XZ_INTERNAL_CRC32

3818 +# include <linux/crc32.h>

3819 +# undef crc32

3820 +# define xz_crc32(buf, size, crc) \

3821 + (~crc32_le(~(uint32_t)(crc), buf, size))

3822 +#endif

3823 +

3824 +/*

3825 + * See the .xz file format specification at

3826 + * http://tukaani.org/xz/xz-file-format.txt

3827 + * to understand the container format.

3828 + */

3829 +

3830 +#define STREAM_HEADER_SIZE 12

3831 +

3832 +#define HEADER_MAGIC "\3757zXZ"

3833 +#define HEADER_MAGIC_SIZE 6

3834 +

3835 +#define FOOTER_MAGIC "YZ"

3836 +#define FOOTER_MAGIC_SIZE 2

3837 +

3838 +/*

3839 + * Variable-length integer can hold a 63-bit unsigned integer or a special

3840 + * value indicating that the value is unknown.

3841 + *

3842 + * Experimental: vli_type can be defined to uint32_t to save a few bytes

3843 + * in code size (no effect on speed). Doing so limits the uncompressed and

3844 + * compressed size of the file to less than 256 MiB and may also weaken

3845 + * error detection slightly.

3846 + */

3847 +typedef uint64_t vli_type;

3848 +

3849 +#define VLI_MAX ((vli_type)-1 / 2)

3850 +#define VLI_UNKNOWN ((vli_type)-1)

3851 +

3852 +/* Maximum encoded size of a VLI */

3853 +#define VLI_BYTES_MAX (sizeof(vli_type) * 8 / 7)

3854 +

3855 +/* Integrity Check types */

3856 +enum xz_check {

3857 + XZ_CHECK_NONE = 0,

3858 + XZ_CHECK_CRC32 = 1,

3859 + XZ_CHECK_CRC64 = 4,

3860 + XZ_CHECK_SHA256 = 10

3861 +};

3862 +

3863 +/* Maximum possible Check ID */

3864 +#define XZ_CHECK_MAX 15

3865 +

3866 +#endif

3867 diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib

3868 index 54fd1b7..b862007 100644

3869 --- a/scripts/Makefile.lib

3870 +++ b/scripts/Makefile.lib

3871 @@ -246,6 +246,34 @@ cmd_lzo = (cat $(filter-out FORCE,$^) | \

3872 lzop -9 && $(call size_append, $(filter-out FORCE,$^))) > $@ || \

3873 (rm -f $@ ; false)

3874

3875 +# XZ

3876 +# ---------------------------------------------------------------------------

3877 +# Use xzkern to compress the kernel image and xzmisc to compress other things.

3878 +#

3879 +# xzkern uses a big LZMA2 dictionary since it doesn't increase memory usage

3880 +# of the kernel decompressor. A BCJ filter is used if it is available for

3881 +# the target architecture. xzkern also appends uncompressed size of the data

3882 +# using size_append. The .xz format has the size information available at

3883 +# the end of the file too, but it's in more complex format and it's good to

3884 +# avoid changing the part of the boot code that reads the uncompressed size.

3885 +# Note that the bytes added by size_append will make the xz tool think that

3886 +# the file is corrupt. This is expected.

3887 +#

3888 +# xzmisc doesn't use size_append, so it can be used to create normal .xz

3889 +# files. xzmisc uses smaller LZMA2 dictionary than xzkern, because a very

3890 +# big dictionary would increase the memory usage too much in the multi-call

3891 +# decompression mode. A BCJ filter isn't used either.

3892 +quiet_cmd_xzkern = XZKERN $@

3893 +cmd_xzkern = (cat $(filter-out FORCE,$^) | \

3894 + sh $(srctree)/scripts/xz_wrap.sh && \

3895 + $(call size_append, $(filter-out FORCE,$^))) > $@ || \

3896 + (rm -f $@ ; false)

3897 +

3898 +quiet_cmd_xzmisc = XZMISC $@

3899 +cmd_xzmisc = (cat $(filter-out FORCE,$^) | \

3900 + xz --check=crc32 --lzma2=dict=1MiB) > $@ || \

3901 + (rm -f $@ ; false)

3902 +

3903 # misc stuff

3904 # ---------------------------------------------------------------------------

3905 quote:="

3906 diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh

3907 new file mode 100644

3908 index 0000000..17a5798

3909 --- /dev/null

3910 +++ b/scripts/xz_wrap.sh

3911 @@ -0,0 +1,23 @@

3912 +#!/bin/sh

3913 +#

3914 +# This is a wrapper for xz to compress the kernel image using appropriate

3915 +# compression options depending on the architecture.

3916 +#

3917 +# Author: Lasse Collin <lasse.collin@tukaani.org>

3918 +#

3919 +# This file has been put into the public domain.

3920 +# You can do whatever you want with this file.

3921 +#

3922 +

3923 +BCJ=

3924 +LZMA2OPTS=

3925 +

3926 +case $ARCH in

3927 + x86|x86_64) BCJ=--x86 ;;

3928 + powerpc) BCJ=--powerpc ;;

3929 + ia64) BCJ=--ia64; LZMA2OPTS=pb=4 ;;

3930 + arm) BCJ=--arm ;;

3931 + sparc) BCJ=--sparc ;;

3932 +esac

3933 +

3934 +exec xz --check=crc32 $BCJ --lzma2=$LZMA2OPTS,dict=32MiB

SliTaz Repositories

wok-tiny view kernel-modular/stuff/001-squashfs-decompressors-add-xz-decompressor-module.patch @ rev 1