Skip to content

fix: skip merging large input sourcemaps#10890

Merged
nicolo-ribaudo merged 1 commit intobabel:masterfrom
JLHwung:skip-merging-large-external-sourcemap
Dec 19, 2019
Merged

fix: skip merging large input sourcemaps#10890
nicolo-ribaudo merged 1 commit intobabel:masterfrom
JLHwung:skip-merging-large-external-sourcemap

Conversation

@JLHwung
Copy link
Copy Markdown
Contributor

@JLHwung JLHwung commented Dec 18, 2019

Q                       A
Fixed Issues? Fixes #10875
Patch: Bug Fix? Yes
Tests Added + Pass? Yes
License MIT

This PR addresses #10875 where a large input source map is consumed and causing OOM error. It is not an ideal fix but still preserve basic support for practical size input source map.

Note that #10885 misidentified the memory bottleneck. But I preserve the change here since we need to detect the source map file size.

Ideally the mergeSourceMap in babel generator should operate in a streaming mode. But this does not fit in a patch release.

I don't consider it as a regression since in 7.6.4 the inputSourceMap is not working properly on external sourcemaps.

@artisb45 @midgethoen Though the URL does not change, here is a new fix. You can download it and overwrite node_modules/@babel/core/lib/transformation/normalize-file.js.

@JLHwung JLHwung added PR: Bug Fix 🐛 A type of pull request used for our changelog categories area: sourcemaps labels Dec 18, 2019
Copy link
Copy Markdown
Member

@nicolo-ribaudo nicolo-ribaudo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we already do something similar in @babel/generator: if the input is >500K (I think) we disable the formatting.

// when `lastComment` is non-null, EXTERNAL_SOURCEMAP_REGEX must have matches
const match: [string, string] = (EXTERNAL_SOURCEMAP_REGEX.exec(
lastComment,
): any);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this guaranteed to never return null?

Copy link
Copy Markdown
Contributor Author

@JLHwung JLHwung Dec 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes because the capturing group

([^\s'"`]+)

has a + quantifier, if the whole regex passed, the capturing group must not be null.

@JLHwung
Copy link
Copy Markdown
Contributor Author

JLHwung commented Dec 18, 2019

@nicolo-ribaudo Correct. I have tested on pdfjs-dist/build/pdf.js.map (1.21 MB) and it does not bail. To be conservative I shrink the threshold to 1M. We can easily change it to an option if people really need to configure.

@nicolo-ribaudo nicolo-ribaudo merged commit e85c9b9 into babel:master Dec 19, 2019
@nicolo-ribaudo nicolo-ribaudo deleted the skip-merging-large-external-sourcemap branch December 19, 2019 00:16
@github-actions github-actions Bot added the outdated A closed issue/PR that is archived due to age. Recommended to make a new issue label Mar 19, 2020
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Mar 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area: sourcemaps outdated A closed issue/PR that is archived due to age. Recommended to make a new issue PR: Bug Fix 🐛 A type of pull request used for our changelog categories

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Very high memory usage compiling mozilla/PDF.js

3 participants