-
-
Notifications
You must be signed in to change notification settings - Fork 35.4k
Expand file tree
/
Copy pathzlib.md
More file actions
694 lines (570 loc) Β· 20.1 KB
/
zlib.md
File metadata and controls
694 lines (570 loc) Β· 20.1 KB
Edit and raw actions
OlderNewer
Β
1
# Zlib
2
3
> Stability: 2 - Stable
4
5
The `zlib` module provides compression functionality implemented using Gzip and
6
Deflate/Inflate. It can be accessed using:
7
8
```js
9
const zlib = require('zlib');
10
```
11
12
Compressing or decompressing a stream (such as a file) can be accomplished by
13
piping the source stream data through a `zlib` stream into a destination stream:
14
15
```js
16
const gzip = zlib.createGzip();
17
const fs = require('fs');
18
const inp = fs.createReadStream('input.txt');
19
const out = fs.createWriteStream('input.txt.gz');
20
21
inp.pipe(gzip).pipe(out);
22
```
23
24
It is also possible to compress or decompress data in a single step:
25
26
```js
27
const input = '.................................';
28
zlib.deflate(input, (err, buffer) => {
29
if (!err) {
30
console.log(buffer.toString('base64'));
31
} else {
32
// handle error
33
}
34
});
35
36
const buffer = Buffer.from('eJzT0yMAAGTvBe8=', 'base64');
37
zlib.unzip(buffer, (err, buffer) => {
38
if (!err) {
39
console.log(buffer.toString());
40
} else {
41
// handle error
42
}
43
});
44
```
45
46
## Compressing HTTP requests and responses
47
48
The `zlib` module can be used to implement support for the `gzip` and `deflate`
49
content-encoding mechanisms defined by
50
[HTTP](https://tools.ietf.org/html/rfc7230#section-4.2).
51
52
The HTTP [`Accept-Encoding`][] header is used within an http request to identify
53
the compression encodings accepted by the client. The [`Content-Encoding`][]
54
header is used to identify the compression encodings actually applied to a
55
message.
56
57
*Note*: the examples given below are drastically simplified to show
58
the basic concept. Using `zlib` encoding can be expensive, and the results
59
ought to be cached. See [Memory Usage Tuning][] for more information
60
on the speed/memory/compression tradeoffs involved in `zlib` usage.
61
62
```js
63
// client request example
64
const zlib = require('zlib');
65
const http = require('http');
66
const fs = require('fs');
67
const request = http.get({ host: 'example.com',
68
path: '/',
69
port: 80,
70
headers: { 'Accept-Encoding': 'gzip,deflate' } });
71
request.on('response', (response) => {
72
const output = fs.createWriteStream('example.com_index.html');
73
74
switch (response.headers['content-encoding']) {
75
// or, just use zlib.createUnzip() to handle both cases
76
case 'gzip':
77
response.pipe(zlib.createGunzip()).pipe(output);
78
break;
79
case 'deflate':
80
response.pipe(zlib.createInflate()).pipe(output);
81
break;
82
default:
83
response.pipe(output);
84
break;
85
}
86
});
87
```
88
89
```js
90
// server example
91
// Running a gzip operation on every request is quite expensive.
92
// It would be much more efficient to cache the compressed buffer.
93
const zlib = require('zlib');
94
const http = require('http');
95
const fs = require('fs');
96
http.createServer((request, response) => {
97
const raw = fs.createReadStream('index.html');
98
let acceptEncoding = request.headers['accept-encoding'];
99
if (!acceptEncoding) {
100
acceptEncoding = '';
101
}
102
103
// Note: This is not a conformant accept-encoding parser.
104
// See http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
105
if (acceptEncoding.match(/\bdeflate\b/)) {
106
response.writeHead(200, { 'Content-Encoding': 'deflate' });
107
raw.pipe(zlib.createDeflate()).pipe(response);
108
} else if (acceptEncoding.match(/\bgzip\b/)) {
109
response.writeHead(200, { 'Content-Encoding': 'gzip' });
110
raw.pipe(zlib.createGzip()).pipe(response);
111
} else {
112
response.writeHead(200, {});
113
raw.pipe(response);
114
}
115
}).listen(1337);
116
```
117
118
By default, the `zlib` methods will throw an error when decompressing
119
truncated data. However, if it is known that the data is incomplete, or
120
the desire is to inspect only the beginning of a compressed file, it is
121
possible to suppress the default error handling by changing the flushing
122
method that is used to compressed the last chunk of input data:
123
124
```js
125
// This is a truncated version of the buffer from the above examples
126
const buffer = Buffer.from('eJzT0yMA', 'base64');
127
128
zlib.unzip(
129
buffer,
130
{finishFlush: zlib.constants.Z_SYNC_FLUSH},
131
(err, buffer) => {
132
if (!err) {
133
console.log(buffer.toString());
134
} else {
135
// handle error
136
}
137
});
138
```
139
140
This will not change the behavior in other error-throwing situations, e.g.
141
when the input data has an invalid format. Using this method, it will not be
142
possible to determine whether the input ended prematurely or lacks the
143
integrity checks, making it necessary to manually check that the
144
decompressed result is valid.
145
146
## Memory Usage Tuning
147
148
<!--type=misc-->
149
150
From `zlib/zconf.h`, modified to node.js's usage:
151
152
The memory requirements for deflate are (in bytes):
153
154
<!-- eslint-disable semi -->
155
```js
156
(1 << (windowBits + 2)) + (1 << (memLevel + 9))
157
```
158
159
That is: 128K for windowBits=15 + 128K for memLevel = 8
160
(default values) plus a few kilobytes for small objects.
161
162
For example, to reduce the default memory requirements from 256K to 128K, the
163
options should be set to:
164
165
```js
166
const options = { windowBits: 14, memLevel: 7 };
167
```
168
169
This will, however, generally degrade compression.
170
171
The memory requirements for inflate are (in bytes) `1 << windowBits`.
172
That is, 32K for windowBits=15 (default value) plus a few kilobytes
173
for small objects.
174
175
This is in addition to a single internal output slab buffer of size
176
`chunkSize`, which defaults to 16K.
177
178
The speed of `zlib` compression is affected most dramatically by the
179
`level` setting. A higher level will result in better compression, but
180
will take longer to complete. A lower level will result in less
181
compression, but will be much faster.
182
183
In general, greater memory usage options will mean that Node.js has to make
184
fewer calls to `zlib` because it will be able to process more data on
185
each `write` operation. So, this is another factor that affects the
186
speed, at the cost of memory usage.
187
188
## Flushing
189
190
Calling [`.flush()`][] on a compression stream will make `zlib` return as much
191
output as currently possible. This may come at the cost of degraded compression
192
quality, but can be useful when data needs to be available as soon as possible.
193
194
In the following example, `flush()` is used to write a compressed partial
195
HTTP response to the client:
196
```js
197
const zlib = require('zlib');
198
const http = require('http');
199
200
http.createServer((request, response) => {
201
// For the sake of simplicity, the Accept-Encoding checks are omitted.
202
response.writeHead(200, { 'content-encoding': 'gzip' });
203
const output = zlib.createGzip();
204
output.pipe(response);
205
206
setInterval(() => {
207
output.write(`The current time is ${Date()}\n`, () => {
208
// The data has been passed to zlib, but the compression algorithm may
209
// have decided to buffer the data for more efficient compression.
210
// Calling .flush() will make the data available as soon as the client
211
// is ready to receive it.
212
output.flush();
213
});
214
}, 1000);
215
}).listen(1337);
216
```
217
218
## Constants
219
<!-- YAML
220
added: v0.5.8
221
-->
222
223
<!--type=misc-->
224
225
All of the constants defined in `zlib.h` are also defined on
226
`require('zlib').constants`. In the normal course of operations, it will not be
227
necessary to use these constants. They are documented so that their presence is
228
not surprising. This section is taken almost directly from the
229
[zlib documentation][]. See <http://zlib.net/manual.html#Constants> for more
230
details.
231
232
*Note*: Previously, the constants were available directly from
233
`require('zlib')`, for instance `zlib.Z_NO_FLUSH`. Accessing the constants
234
directly from the module is currently still possible but should be considered
235
deprecated.
236
237
Allowed flush values.
238
239
* `zlib.constants.Z_NO_FLUSH`
240
* `zlib.constants.Z_PARTIAL_FLUSH`
241
* `zlib.constants.Z_SYNC_FLUSH`
242
* `zlib.constants.Z_FULL_FLUSH`
243
* `zlib.constants.Z_FINISH`
244
* `zlib.constants.Z_BLOCK`
245
* `zlib.constants.Z_TREES`
246
247
Return codes for the compression/decompression functions. Negative
248
values are errors, positive values are used for special but normal
249
events.
250
251
* `zlib.constants.Z_OK`
252
* `zlib.constants.Z_STREAM_END`
253
* `zlib.constants.Z_NEED_DICT`
254
* `zlib.constants.Z_ERRNO`
255
* `zlib.constants.Z_STREAM_ERROR`
256
* `zlib.constants.Z_DATA_ERROR`
257
* `zlib.constants.Z_MEM_ERROR`
258
* `zlib.constants.Z_BUF_ERROR`
259
* `zlib.constants.Z_VERSION_ERROR`
260
261
Compression levels.
262
263
* `zlib.constants.Z_NO_COMPRESSION`
264
* `zlib.constants.Z_BEST_SPEED`
265
* `zlib.constants.Z_BEST_COMPRESSION`
266
* `zlib.constants.Z_DEFAULT_COMPRESSION`
267
268
Compression strategy.
269
270
* `zlib.constants.Z_FILTERED`
271
* `zlib.constants.Z_HUFFMAN_ONLY`
272
* `zlib.constants.Z_RLE`
273
* `zlib.constants.Z_FIXED`
274
* `zlib.constants.Z_DEFAULT_STRATEGY`
275
276
## Class Options
277
<!-- YAML
278
added: v0.11.1
279
changes:
280
- version: v8.0.0
281
pr-url: https://github.com/nodejs/node/pull/12001
282
description: The `dictionary` option can be an Uint8Array now.
283
- version: v5.11.0
284
pr-url: https://github.com/nodejs/node/pull/6069
285
description: The `finishFlush` option is supported now.
286
-->
287
288
<!--type=misc-->
289
290
Each class takes an `options` object. All options are optional.
291
292
Note that some options are only relevant when compressing, and are
293
ignored by the decompression classes.
294
295
* `flush` {integer} (default: `zlib.constants.Z_NO_FLUSH`)
296
* `finishFlush` {integer} (default: `zlib.constants.Z_FINISH`)
297
* `chunkSize` {integer} (default: 16\*1024)
298
* `windowBits` {integer}
299
* `level` {integer} (compression only)
300
* `memLevel` {integer} (compression only)
301
* `strategy` {integer} (compression only)
302
* `dictionary` {Buffer|TypedArray|DataView} (deflate/inflate only, empty dictionary by
303
default)
304
305
See the description of `deflateInit2` and `inflateInit2` at
306
<http://zlib.net/manual.html#Advanced> for more information on these.
307
308
## Class: zlib.Deflate
309
<!-- YAML
310
added: v0.5.8
311
-->
312
313
Compress data using deflate.
314
315
## Class: zlib.DeflateRaw
316
<!-- YAML
317
added: v0.5.8
318
-->
319
320
Compress data using deflate, and do not append a `zlib` header.
321
322
## Class: zlib.Gunzip
323
<!-- YAML
324
added: v0.5.8
325
changes:
326
- version: v6.0.0
327
pr-url: https://github.com/nodejs/node/pull/5883
328
description: Trailing garbage at the end of the input stream will now
329
result in an `error` event.
330
- version: v5.9.0
331
pr-url: https://github.com/nodejs/node/pull/5120
332
description: Multiple concatenated gzip file members are supported now.
333
- version: v5.0.0
334
pr-url: https://github.com/nodejs/node/pull/2595
335
description: A truncated input stream will now result in an `error` event.
336
-->
337
338
Decompress a gzip stream.
339
340
## Class: zlib.Gzip
341
<!-- YAML
342
added: v0.5.8
343
-->
344
345
Compress data using gzip.
346
347
## Class: zlib.Inflate
348
<!-- YAML
349
added: v0.5.8
350
changes:
351
- version: v5.0.0
352
pr-url: https://github.com/nodejs/node/pull/2595
353
description: A truncated input stream will now result in an `error` event.
354
-->
355
356
Decompress a deflate stream.
357
358
## Class: zlib.InflateRaw
359
<!-- YAML
360
added: v0.5.8
361
changes:
362
- version: v6.8.0
363
pr-url: https://github.com/nodejs/node/pull/8512
364
description: Custom dictionaries are now supported by `InflateRaw`.
365
- version: v5.0.0
366
pr-url: https://github.com/nodejs/node/pull/2595
367
description: A truncated input stream will now result in an `error` event.
368
-->
369
370
Decompress a raw deflate stream.
371
372
## Class: zlib.Unzip
373
<!-- YAML
374
added: v0.5.8
375
-->
376
377
Decompress either a Gzip- or Deflate-compressed stream by auto-detecting
378
the header.
379
380
## Class: zlib.Zlib
381
<!-- YAML
382
added: v0.5.8
383
-->
384
385
Not exported by the `zlib` module. It is documented here because it is the base
386
class of the compressor/decompressor classes.
387
388
### zlib.flush([kind], callback)
389
<!-- YAML
390
added: v0.5.8
391
-->
392
393
`kind` defaults to `zlib.constants.Z_FULL_FLUSH`.
394
395
Flush pending data. Don't call this frivolously, premature flushes negatively
396
impact the effectiveness of the compression algorithm.
397
398
Calling this only flushes data from the internal `zlib` state, and does not
399
perform flushing of any kind on the streams level. Rather, it behaves like a
400
normal call to `.write()`, i.e. it will be queued up behind other pending
401
writes and will only produce output when data is being read from the stream.
402
403
### zlib.params(level, strategy, callback)
404
<!-- YAML
405
added: v0.11.4
406
-->
407
408
Dynamically update the compression level and compression strategy.
409
Only applicable to deflate algorithm.
410
411
### zlib.reset()
412
<!-- YAML
413
added: v0.7.0
414
-->
415
416
Reset the compressor/decompressor to factory defaults. Only applicable to
417
the inflate and deflate algorithms.
418
419
## zlib.constants
420
<!-- YAML
421
added: v7.0.0
422
-->
423
424
Provides an object enumerating Zlib-related constants.
425
426
## zlib.createDeflate([options])
427
<!-- YAML
428
added: v0.5.8
429
-->
430
431
Returns a new [Deflate][] object with an [options][].
432
433
## zlib.createDeflateRaw([options])
434
<!-- YAML
435
added: v0.5.8
436
-->
437
438
Returns a new [DeflateRaw][] object with an [options][].
439
440
*Note*: The zlib library rejects requests for 256-byte windows (i.e.,
441
`{ windowBits: 8 }` in `options`). An `Error` will be thrown when creating
442
a [DeflateRaw][] object with this specific value of the `windowBits` option.
443
444
## zlib.createGunzip([options])
445
<!-- YAML
446
added: v0.5.8
447
-->
448
449
Returns a new [Gunzip][] object with an [options][].
450
451
## zlib.createGzip([options])
452
<!-- YAML
453
added: v0.5.8
454
-->
455
456
Returns a new [Gzip][] object with an [options][].
457
458
## zlib.createInflate([options])
459
<!-- YAML
460
added: v0.5.8
461
-->
462
463
Returns a new [Inflate][] object with an [options][].
464
465
## zlib.createInflateRaw([options])
466
<!-- YAML
467
added: v0.5.8
468
-->
469
470
Returns a new [InflateRaw][] object with an [options][].
471
472
## zlib.createUnzip([options])
473
<!-- YAML
474
added: v0.5.8
475
-->
476
477
Returns a new [Unzip][] object with an [options][].
478
479
## Convenience Methods
480
481
<!--type=misc-->
482
483
All of these take a [`Buffer`][], [`TypedArray`][], [`DataView`][], or string as
484
the first argument, an optional second argument to supply options to the `zlib`
485
classes and will call the supplied callback with `callback(error, result)`.
486
487
Every method has a `*Sync` counterpart, which accept the same arguments, but
488
without a callback.
489
490
### zlib.deflate(buffer[, options], callback)
491
<!-- YAML
492
added: v0.6.0
493
changes:
494
- version: v8.0.0
495
pr-url: https://github.com/nodejs/node/pull/12223
496
description: The `buffer` parameter can be any TypedArray or DataView now.
497
- version: v8.0.0
498
pr-url: https://github.com/nodejs/node/pull/12001
499
description: The `buffer` parameter can be an Uint8Array now.
500
-->
501
### zlib.deflateSync(buffer[, options])
502
<!-- YAML
503
added: v0.11.12
504
changes:
505
- version: v8.0.0
506
pr-url: https://github.com/nodejs/node/pull/12223
507
description: The `buffer` parameter can be any TypedArray or DataView now.
508
- version: v8.0.0
509
pr-url: https://github.com/nodejs/node/pull/12001
510
description: The `buffer` parameter can be an Uint8Array now.
511
-->
512
513
- `buffer` {Buffer|TypedArray|DataView|string}
514
515
Compress a chunk of data with [Deflate][].
516
517
### zlib.deflateRaw(buffer[, options], callback)
518
<!-- YAML
519
added: v0.6.0
520
changes:
521
- version: v8.0.0
522
pr-url: https://github.com/nodejs/node/pull/12223
523
description: The `buffer` parameter can be any TypedArray or DataView now.
524
- version: v8.0.0
525
pr-url: https://github.com/nodejs/node/pull/12001
526
description: The `buffer` parameter can be an Uint8Array now.
527
-->
528
### zlib.deflateRawSync(buffer[, options])
529
<!-- YAML
530
added: v0.11.12
531
changes:
532
- version: v8.0.0
533
pr-url: https://github.com/nodejs/node/pull/12223
534
description: The `buffer` parameter can be any TypedArray or DataView now.
535
- version: v8.0.0
536
pr-url: https://github.com/nodejs/node/pull/12001
537
description: The `buffer` parameter can be an Uint8Array now.
538
-->
539
540
- `buffer` {Buffer|TypedArray|DataView|string}
541
542
Compress a chunk of data with [DeflateRaw][].
543
544
### zlib.gunzip(buffer[, options], callback)
545
<!-- YAML
546
added: v0.6.0
547
changes:
548
- version: v8.0.0
549
pr-url: https://github.com/nodejs/node/pull/12223
550
description: The `buffer` parameter can be any TypedArray or DataView now.
551
- version: v8.0.0
552
pr-url: https://github.com/nodejs/node/pull/12001
553
description: The `buffer` parameter can be an Uint8Array now.
554
-->
555
### zlib.gunzipSync(buffer[, options])
556
<!-- YAML
557
added: v0.11.12
558
changes:
559
- version: v8.0.0
560
pr-url: https://github.com/nodejs/node/pull/12223
561
description: The `buffer` parameter can be any TypedArray or DataView now.
562
- version: v8.0.0
563
pr-url: https://github.com/nodejs/node/pull/12001
564
description: The `buffer` parameter can be an Uint8Array now.
565
-->
566
567
- `buffer` {Buffer|TypedArray|DataView|string}
568
569
Decompress a chunk of data with [Gunzip][].
570
571
### zlib.gzip(buffer[, options], callback)
572
<!-- YAML
573
added: v0.6.0
574
changes:
575
- version: v8.0.0
576
pr-url: https://github.com/nodejs/node/pull/12223
577
description: The `buffer` parameter can be any TypedArray or DataView now.
578
- version: v8.0.0
579
pr-url: https://github.com/nodejs/node/pull/12001
580
description: The `buffer` parameter can be an Uint8Array now.
581
-->
582
### zlib.gzipSync(buffer[, options])
583
<!-- YAML
584
added: v0.11.12
585
changes:
586
- version: v8.0.0
587
pr-url: https://github.com/nodejs/node/pull/12223
588
description: The `buffer` parameter can be any TypedArray or DataView now.
589
- version: v8.0.0
590
pr-url: https://github.com/nodejs/node/pull/12001
591
description: The `buffer` parameter can be an Uint8Array now.
592
-->
593
594
- `buffer` {Buffer|TypedArray|DataView|string}
595
596
Compress a chunk of data with [Gzip][].
597
598
### zlib.inflate(buffer[, options], callback)
599
<!-- YAML
600
added: v0.6.0
601
changes:
602
- version: v8.0.0
603
pr-url: https://github.com/nodejs/node/pull/12223
604
description: The `buffer` parameter can be any TypedArray or DataView now.
605
- version: v8.0.0
606
pr-url: https://github.com/nodejs/node/pull/12001
607
description: The `buffer` parameter can be an Uint8Array now.
608
-->
609
### zlib.inflateSync(buffer[, options])
610
<!-- YAML
611
added: v0.11.12
612
changes:
613
- version: v8.0.0
614
pr-url: https://github.com/nodejs/node/pull/12223
615
description: The `buffer` parameter can be any TypedArray or DataView now.
616
- version: v8.0.0
617
pr-url: https://github.com/nodejs/node/pull/12001
618
description: The `buffer` parameter can be an Uint8Array now.
619
-->
620
621
- `buffer` {Buffer|TypedArray|DataView|string}
622
623
Decompress a chunk of data with [Inflate][].
624
625
### zlib.inflateRaw(buffer[, options], callback)
626
<!-- YAML
627
added: v0.6.0
628
changes:
629
- version: v8.0.0
630
pr-url: https://github.com/nodejs/node/pull/12223
631
description: The `buffer` parameter can be any TypedArray or DataView now.
632
- version: v8.0.0
633
pr-url: https://github.com/nodejs/node/pull/12001
634
description: The `buffer` parameter can be an Uint8Array now.
635
-->
636
### zlib.inflateRawSync(buffer[, options])
637
<!-- YAML
638
added: v0.11.12
639
changes:
640
- version: v8.0.0
641
pr-url: https://github.com/nodejs/node/pull/12223
642
description: The `buffer` parameter can be any TypedArray or DataView now.
643
- version: v8.0.0
644
pr-url: https://github.com/nodejs/node/pull/12001
645
description: The `buffer` parameter can be an Uint8Array now.
646
-->
647
648
- `buffer` {Buffer|TypedArray|DataView|string}
649
650
Decompress a chunk of data with [InflateRaw][].
651
652
### zlib.unzip(buffer[, options], callback)
653
<!-- YAML
654
added: v0.6.0
655
changes:
656
- version: v8.0.0
657
pr-url: https://github.com/nodejs/node/pull/12223
658
description: The `buffer` parameter can be any TypedArray or DataView now.
659
- version: v8.0.0
660
pr-url: https://github.com/nodejs/node/pull/12001
661
description: The `buffer` parameter can be an Uint8Array now.
662
-->
663
### zlib.unzipSync(buffer[, options])
664
<!-- YAML
665
added: v0.11.12
666
changes:
667
- version: v8.0.0
668
pr-url: https://github.com/nodejs/node/pull/12223
669
description: The `buffer` parameter can be any TypedArray or DataView now.
670
- version: v8.0.0
671
pr-url: https://github.com/nodejs/node/pull/12001
672
description: The `buffer` parameter can be an Uint8Array now.
673
-->
674
675
- `buffer` {Buffer|TypedArray|DataView|string}
676
677
Decompress a chunk of data with [Unzip][].
678
679
[`.flush()`]: #zlib_zlib_flush_kind_callback
680
[`Accept-Encoding`]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
681
[`Buffer`]: buffer.html#buffer_class_buffer
682
[`Content-Encoding`]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11
683
[`DataView`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView
684
[`TypedArray`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray
685
[DeflateRaw]: #zlib_class_zlib_deflateraw
686
[Deflate]: #zlib_class_zlib_deflate
687
[Gunzip]: #zlib_class_zlib_gunzip
688
[Gzip]: #zlib_class_zlib_gzip
689
[InflateRaw]: #zlib_class_zlib_inflateraw
690
[Inflate]: #zlib_class_zlib_inflate
691
[Memory Usage Tuning]: #zlib_memory_usage_tuning
692
[Unzip]: #zlib_class_zlib_unzip
693
[options]: #zlib_class_options
694
[zlib documentation]: http://zlib.net/manual.html#Constants