Skip to content

test: use normalize() for unicode paths - #3007

Closed
silverwind wants to merge 0 commit into
nodejs:masterfrom
silverwind:normalize-fix
Closed

test: use normalize() for unicode paths#3007
silverwind wants to merge 0 commit into
nodejs:masterfrom
silverwind:normalize-fix

Conversation

@silverwind

Copy link
Copy Markdown
Contributor

Went ahead and changed the failing test to use String.prototype.normalize so path strings are compared form-independently.

@srl295 The test will still fail on debug builds that don't include ICU. Is there a way I can feature-test if the current build includes ICU? Maybe we should add process.versions.icu?

Related: #2165
Docs Issue: nodejs/docs#42

@silverwind silverwind added test Issues and PRs related to Node.js core tests and test infrastructure. macos Issues and PRs related to the macOS platform. i18n-api Issues and PRs related to Node.js internationalization support. labels Sep 22, 2015
@silverwind

Copy link
Copy Markdown
Contributor Author

CI: https://ci.nodejs.org/job/node-test-pull-request/362/

(Let's hope CI builds have ICU enabled)

@srl295

srl295 commented Sep 22, 2015

Copy link
Copy Markdown
Member

@silverwind they have small ICU enabled, because that's the default since v3.1.0.

See https://github.com/nodejs/node/blob/master/test/parallel/test-intl.js#L13 for an example of testing whether ICU was enabled.

@silverwind

Copy link
Copy Markdown
Contributor Author

@srl295 thanks. Can one assume that both the root and en locales will always be present when built with ICU? What's the root locale exactly, by the way?

@silverwind

Copy link
Copy Markdown
Contributor Author

Updated the test so it works on non-ICU builds too by using characters that can't be decomposed (emoji) when ICU is not present.

Another CI: https://ci.nodejs.org/job/node-test-pull-request/365/

@srl295

srl295 commented Sep 25, 2015

Copy link
Copy Markdown
Member

@silverwind no.. you can't really assume it. but see process.config.variables.icu_locales

 > process.config.variables.icu_locales.split(',')
 [ 'en', 'root' ]

Root is just the "base" locale ID. It should have been named und if iso-639-2 had been around way back when. It's not English, but its collation is (close to) UCA, and it contains items that are common to other locales.

@silverwind

Copy link
Copy Markdown
Contributor Author

I will rework this once #3089 is landed.

@silverwind

Copy link
Copy Markdown
Contributor Author

Using process.version.icu now. Any feedback?

@jbergstroem

Copy link
Copy Markdown
Member

LGTM

@silverwind

Copy link
Copy Markdown
Contributor Author

There's a slight issue here: the test change depends on process.versions.icu which will land in 4.2, while this test change could go in earlier in 4.1.x. Won't be an issue on master thought.

cc: @nodejs/release

@silverwind

Copy link
Copy Markdown
Contributor Author

It won't fail dramatically in 4.1.x thought, it just won't actually test .normalize, even if Intl is there, so might be a non-issue.

@rvagg

rvagg commented Oct 3, 2015

Copy link
Copy Markdown
Member

might leave it out of 4.1.2 eh? don't want the commit suggesting to people that it's actually doing something

@silverwind

Copy link
Copy Markdown
Contributor Author

Alright, will wait after 4.1.2 with landing this.

@Fishrock123

Copy link
Copy Markdown
Contributor

LGTM if CI is happy and others sign-off.

@silverwind

Copy link
Copy Markdown
Contributor Author

One more CI with the latest change: https://ci.nodejs.org/job/node-test-pull-request/418/

@thefourtheye

Copy link
Copy Markdown
Contributor

Change LGTM, hope it fixes the failure.

silverwind added a commit that referenced this pull request Oct 6, 2015
OS X 10.11 changed the unicode normalization form of certain code points
returned by system calls like getcwd() from NFC to NFD which made
results in this test failing.

The consensus of #2165 is to delegate
the task of unicode normalization to the user, and work will continue to
document how to handle unicode in a form-sensitive file system.

PR-URL: #3007
Fixes: #2165
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
@silverwind silverwind closed this Oct 6, 2015
@silverwind
silverwind deleted the normalize-fix branch October 6, 2015 16:14
@silverwind

Copy link
Copy Markdown
Contributor Author

Landed in 81e98e9, thanks!

@Fishrock123

Copy link
Copy Markdown
Contributor

@silverwind this requires ICU right? Does this test still work properly without it?

Fwiw, the CI compiles without ICU for test runs.

@silverwind

Copy link
Copy Markdown
Contributor Author

@Fishrock123 it works in non-ICU builds too, where process.version.icu will be undefined, and in which case we write an emoji-filename which can't be decomposed 😉.

@silverwind

Copy link
Copy Markdown
Contributor Author

Also, .normalize() returns the string as-is in non-ICU builds, if you're wondering.

@jasnell jasnell mentioned this pull request Oct 8, 2015
29 tasks
silverwind added a commit that referenced this pull request Oct 8, 2015
OS X 10.11 changed the unicode normalization form of certain code points
returned by system calls like getcwd() from NFC to NFD which made
results in this test failing.

The consensus of #2165 is to delegate
the task of unicode normalization to the user, and work will continue to
document how to handle unicode in a form-sensitive file system.

PR-URL: #3007
Fixes: #2165
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

i18n-api Issues and PRs related to Node.js internationalization support. macos Issues and PRs related to the macOS platform. test Issues and PRs related to Node.js core tests and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants