readline: add unicode line separator option#63789
Conversation
Signed-off-by: Scarab Systems <scarab.systems@yahoo.com>
90cb9fa to
d936651
Compare
| * `unicodeLineSeparators` {boolean} If `true`, `\u2028` and `\u2029` will be | ||
| treated as end-of-line input. **Default:** `true`. |
There was a problem hiding this comment.
I wonder if a boolean is the correct way, or if we should instead let the user define their own list of separator as suggested in #60606 (comment)
There was a problem hiding this comment.
That makes sense. I agree a boolean may be too narrow if we are adding a new API surface.
Would an additionalLineSeparators option be a better shape here? My thinking is that \n, \r, and \r\n should remain the fixed readline line endings, while this option controls only the extra separators. The default could preserve current behavior with ['\u2028', '\u2029'], and JSONL users could pass [] to preserve U+2028/U+2029 inside records.
I can rework the PR in that direction if that seems aligned with what you had in mind.
There was a problem hiding this comment.
I agree that reverting back to only \n/\r is likely a goal, though that would be a semver change, so we'd better land that separately. We could add a option lineSeparators that would take either a string, a regex, and maybe some magic values (e.g. Symbol.for('nodejs.readline.classic'), Symbol.for('nodejs.readline.unicode'))
Fixes: #22448
This adds a
unicodeLineSeparatorsoption toreadline.createInterface()andreadlinePromises.createInterface().By default, behavior is unchanged:
\u2028and\u2029continue to be treated as line endings. WhenunicodeLineSeparators: falseis passed, readline only splits on CR, LF, and CRLF, allowing formats such as JSONL to preserve Unicode line and paragraph separators inside record contents.Tests added/updated:
./node test/parallel/test-readline-line-separators.jspython3 tools/test.py --shell ./node test/parallel/test-readline-line-separators.jsmake lint-js LINT_JS_TARGETS="lib/internal/readline/interface.js lib/readline.js test/parallel/test-readline-line-separators.js"make lint-md LINT_MD_FILES="doc/api/readline.md"