Skip to content

doc: add documentation for invalid byte sequences #28249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rexagod wants to merge 6 commits into from
Closed

doc: add documentation for invalid byte sequences #28249

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
aa01448
doc: add documentation for invalid byte sequences
rexagod Jun 16, 2019
a5c9915
Update buffer.md
rexagod Jun 16, 2019
0b2d591
re-position doc segment
rexagod Jun 16, 2019
1739d22
Update doc/api/buffer.md
rexagod Jul 5, 2019
834df54
Update doc/api/buffer.md
rexagod Jul 5, 2019
3e7d1bb
Update doc/api/buffer.md
rexagod Jul 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
re-position doc segment
  • Loading branch information
@rexagod
rexagod authored Jun 16, 2019
commit 0b2d59101e119a398b56438de763f55454ecfa72
64 changes: 32 additions & 32 deletions doc/api/buffer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,38 +165,6 @@ console.log(Buffer.from('fhqwhgads', 'utf16le'));
// Prints: <Buffer 66 00 68 00 71 00 77 00 68 00 67 00 61 00 64 00 73 00>
```

### Evaluating legal code points for '`utf-8'` encoding

Byte sequences that do not have corresponding UTF-16 encodings and non-legal
Unicode values, along with their UTF-8 counterparts must be treated as
invalid byte sequences.

For cases regarding operations other than employing backward compatibility
for 7-bit (and [extended 8-bit]((https://en.wikipedia.org/wiki/UTF-8#Description))
in rare cases) `'ascii'` data, and the valid [`UTF-8` code units](https://en.wikipedia.org/wiki/UTF-8#Codepage_layout),
it should be noted that the replacement character (`�`) is returned,
and *no exception will be thrown*.

It should also be noted that a `U+FFFD` replacement value
(representing the aforementioned replacement character) will be returned
in case of decoding errors (invalid unicode scalar values).

```js
// Assuming an invalid byte sequence
const buf = Buffer.from([237, 166, 164]);

const buf_str = buf.toString('utf-8');

console.log(buf_str);
// Prints: '�'

console.log(buf.byteLength(buf_str));
// Prints: 3

console.log(buf.codePointAt(0).toString(16));
// Prints: 'fffd'
```

The character encodings currently supported by Node.js include:

* `'ascii'` - For 7-bit ASCII data only. This encoding is fast and will strip
Expand Down Expand Up @@ -229,6 +197,38 @@ the WHATWG specification it is possible that the server actually returned
`'win-1252'`-encoded data, and using `'latin1'` encoding may incorrectly decode
the characters.

### Evaluating legal code points for '`utf-8'` encoding

Byte sequences that do not have corresponding UTF-16 encodings and non-legal
Unicode values, along with their UTF-8 counterparts must be treated as
invalid byte sequences.

For cases regarding operations other than employing backward compatibility
for 7-bit (and [extended 8-bit]((https://en.wikipedia.org/wiki/UTF-8#Description))
in rare cases) `'ascii'` data, and the valid [`UTF-8` code units](https://en.wikipedia.org/wiki/UTF-8#Codepage_layout),
it should be noted that the replacement character (`�`) is returned,
Comment thread
rexagod marked this conversation as resolved.
Outdated
and *no exception will be thrown*.
Comment thread
rexagod marked this conversation as resolved.
Outdated

It should also be noted that a `U+FFFD` replacement value
Comment thread
rexagod marked this conversation as resolved.
Outdated
(representing the aforementioned replacement character) will be returned
in case of decoding errors (invalid unicode scalar values).
Copy link
Copy Markdown
Member

@addaleax addaleax Jul 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I don’t understand most of the text or its relevance here… the text basically says that invalid UTF-8 byte sequences will be decoded into U+FFFD replacement characters and that no error will be thrown in those cases, right?

How do UTF-16 and ASCII relate to that? What does “non-legal Unicode value” mean? (I would guess that this refers to characters that would be beyond U+10FFFF – if that’s correct, can you clarify that in the text?)


```js
// Assuming an invalid byte sequence
const buf = Buffer.from([237, 166, 164]);

const buf_str = buf.toString('utf-8');

console.log(buf_str);
// Prints: '�'

console.log(buf.byteLength(buf_str));
// Prints: 3

console.log(buf.codePointAt(0).toString(16));
// Prints: 'fffd'
```

## Buffers and TypedArray
<!-- YAML
changes:
Expand Down