Skip to content

zlib: expose amount of data read for engines #13088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AlexanderOMara wants to merge 5 commits into from
Closed

zlib: expose amount of data read for engines #13088

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7aa6e2b
zlib: expose amount of data read for engines
AlexanderOMara May 18, 2017
df6c1f0
zlib: document engine bytesRead property
AlexanderOMara May 22, 2017
cf58570
zlib: refactored bytesRead test to use callbacks
AlexanderOMara May 24, 2017
f5a87d6
zlib: expanded bytesRead property description
AlexanderOMara May 24, 2017
6c1a1e9
zlib: added bytesRead YAML comment
AlexanderOMara May 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions doc/api/zlib.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,12 @@ added: v0.5.8
Not exported by the `zlib` module. It is documented here because it is the base
class of the compressor/decompressor classes.

### zlib.bytesRead
Copy link
Copy Markdown
Member

@addaleax addaleax May 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add

<!-- YAML
added: REPLACEME
-->

directly below this heading?

Copy link
Copy Markdown
Contributor Author

@AlexanderOMara AlexanderOMara May 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added


* {number}

The `zlib.bytesRead` property specifies the number of bytes read by the engine.
Copy link
Copy Markdown
Contributor

@sam-github sam-github May 23, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bytes in -> zlib engine -> bytes out: this is the bytes read by the engine from the encrypted stream (bytes out), or the other side (bytes in)? I suspect its bytes in, but it should be overwhelmingly clear from the docs, please, perhaps "bytes read by the engine before the bytes are processed (compressed or decompressed, as appropriate for the derived class)" <--- or change the language if its bytes read post processing.

Copy link
Copy Markdown
Contributor Author

@AlexanderOMara AlexanderOMara May 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's the bytes in. Description expanded.


### zlib.flush([kind], callback)
<!-- YAML
added: v0.5.8
Expand Down
4 changes: 4 additions & 0 deletions lib/zlib.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,8 @@ class Zlib extends Transform {
opts = opts || {};
super(opts);

this.bytesRead = 0;

this._opts = opts;
this._chunkSize = opts.chunkSize || constants.Z_DEFAULT_CHUNK;

Expand Down Expand Up @@ -408,6 +410,8 @@ class Zlib extends Transform {
var have = availOutBefore - availOutAfter;
assert(have >= 0, 'have should not go down');

self.bytesRead += availInBefore - availInAfter;

if (have > 0) {
var out = self._buffer.slice(self._offset, self._offset + have);
self._offset += have;
Expand Down
92 changes: 92 additions & 0 deletions test/parallel/test-zlib-bytes-read.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
'use strict';
const common = require('../common');
const assert = require('assert');
const zlib = require('zlib');

const expectStr = 'abcdefghijklmnopqrstuvwxyz'.repeat(2);
const expectBuf = Buffer.from(expectStr);

function createWriter(target, buffer) {
const writer = { size: 0 };
const write = () => {
target.write(Buffer.from([buffer[writer.size++]]));
if (writer.size < buffer.length) {
setTimeout(write, 25);
Copy link
Copy Markdown
Member

@addaleax addaleax May 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks a bit odd… I think what you want is to do something once the write finished, but the right way to do that would be to pass a callback to target.write(), instead of relying on timeouts

Copy link
Copy Markdown
Contributor Author

@AlexanderOMara AlexanderOMara May 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't realize there were callbacks for these methods. That's a much better idea.

} else {
target.end();
}
};
write();
return writer;
}

for (const method of [
['createGzip', 'createGunzip', false],
['createGzip', 'createUnzip', false],
['createDeflate', 'createInflate', true],
['createDeflateRaw', 'createInflateRaw', true]
]) {
let compWriter;
let compData = new Buffer(0);

const comp = zlib[method[0]]();
comp.on('data', function(d) {
compData = Buffer.concat([compData, d]);
assert.strictEqual(this.bytesRead, compWriter.size,
`Should get write size on ${method[0]} data.`);
});
comp.on('end', common.mustCall(function() {
assert.strictEqual(this.bytesRead, compWriter.size,
`Should get write size on ${method[0]} end.`);
assert.strictEqual(this.bytesRead, expectStr.length,
`Should get data size on ${method[0]} end.`);

{
let decompWriter;
let decompData = new Buffer(0);

const decomp = zlib[method[1]]();
decomp.on('data', function(d) {
decompData = Buffer.concat([decompData, d]);
assert.strictEqual(this.bytesRead, decompWriter.size,
`Should get write size on ${method[0]}/` +
`${method[1]} data.`);
});
decomp.on('end', common.mustCall(function() {
assert.strictEqual(this.bytesRead, compData.length,
`Should get compressed size on ${method[0]}/` +
`${method[1]} end.`);
assert.strictEqual(decompData.toString(), expectStr,
`Should get original string on ${method[0]}/` +
`${method[1]} end.`);
}));
decompWriter = createWriter(decomp, compData);
}

// Some methods should allow extra data after the compressed data
if (method[2]) {
const compDataExtra = Buffer.concat([compData, new Buffer('extra')]);

let decompWriter;
let decompData = new Buffer(0);

const decomp = zlib[method[1]]();
decomp.on('data', function(d) {
decompData = Buffer.concat([decompData, d]);
assert.strictEqual(this.bytesRead, decompWriter.size,
`Should get write size on ${method[0]}/` +
`${method[1]} data.`);
});
decomp.on('end', common.mustCall(function() {
assert.strictEqual(this.bytesRead, compData.length,
`Should get compressed size on ${method[0]}/` +
`${method[1]} end.`);
assert.strictEqual(decompData.toString(), expectStr,
`Should get original string on ${method[0]}/` +
`${method[1]} end.`);
}));
decompWriter = createWriter(decomp, compDataExtra);
}
}));
compWriter = createWriter(comp, expectBuf);
}