Add Lavamoat to build system by EtDu · Pull Request #9939 · MetaMask/metamask-extension

EtDu · 2020-11-24T07:08:56Z

This PR aims to add LavaMoat in the MetaMask build system.

Introduces patch-package to work around SES compatibility issues in deep dependencies
Add explicit imports for babel plugins and presets
Browserify: use explicit require statements to import plugins & transforms
Browserify: use explicit imports for transforms in dependencies (required for static analysis)
Remove redundant gulp-sass assignment to compiler

Regular config modifications:

Fix deep endowments bug: remove .match from process.version
Remove .call from hasOwnProperty & stream.Stream
Lavamoat fails to dedupe configuration when merging override configuration (E.G. x.y & x.y.z)

Override config additions:

Endow babel presets & plugins manually (dynamic require)
Endow to @babel/core for reading babel config
Endow Browerify transform loose-envify
Endow post-css iterations to stylelint (lazy-import)
incomplete parsing: vinyl, colorette, node-sass

github-actions · 2020-11-24T07:09:10Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

metamaskbot · 2020-11-24T07:33:22Z

Builds ready [d6aacf5]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (463 ± 45 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	30	96	54	21	10
		domContentLoaded	302	586	461	93	44
		load	303	587	463	93	45
		domInteractive	302	586	461	93	44

metamaskbot · 2020-12-01T04:24:10Z

Builds ready [111772b]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (539 ± 49 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	33	112	56	23	11
		domContentLoaded	341	662	537	101	48
		load	342	663	539	101	49
		domInteractive	341	661	536	101	48

metamaskbot · 2020-12-07T04:23:23Z

Builds ready [8dbc38a]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
dep viz: background
all artifacts

Page Load Metrics (463 ± 54 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	39	100	68	18	9
		domContentLoaded	328	656	461	113	54
		load	330	658	463	113	54
		domInteractive	328	655	460	113	54

metamaskbot · 2020-12-17T05:59:25Z

Builds ready [45b3e50]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
all artifacts

Page Load Metrics (414 ± 48 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	31	347	69	66	32
		domContentLoaded	262	604	412	100	48
		load	264	607	414	101	48
		domInteractive	262	604	411	100	48

metamaskbot · 2020-12-17T06:24:02Z

Builds ready [96b8ee4]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
code coverage: Report
all artifacts

Page Load Metrics (595 ± 46 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	33	76	51	12	6
		domContentLoaded	346	825	593	96	46
		load	348	826	595	96	46
		domInteractive	346	824	593	96	46

metamaskbot · 2020-12-17T07:21:44Z

Builds ready [dcc16e5]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
code coverage: Report
all artifacts

Page Load Metrics (516 ± 14 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	31	62	47	9	4
		domContentLoaded	398	541	513	29	14
		load	399	543	516	30	14
		domInteractive	397	541	513	29	14

kumavis · 2020-12-17T07:51:27Z

wow nice, this is ready

kumavis · 2020-12-17T07:52:55Z

this pr is ready but i recommend waiting til after the holiday

Gudahtt · 2021-01-23T18:57:48Z

Now that v9 is out, I think we're ready to start trying to land this. Could you rebase this on develop?

kumavis · 2021-02-08T08:39:04Z

🦇 🦇 🦇 🦇 🦇 🦇 🦇 🦇 🦇 🦇

build completed. task timeline:
█ clean NaNs
█ styles:prod NaNs
█ manifest:prod NaNs
█ scripts:core:prod:disable-console NaNs
█ scripts:core:prod:initSentry NaNs
█ scripts:core:prod:phishing-detect NaNs
█ scripts:core:prod:contentscript NaNs
█ scripts:deps:ui NaNs
█ scripts:deps:background NaNs
█ static:prod NaNs
█ scripts:core:prod:background NaNs
█ scripts:core:prod:ui NaNs
█ zip NaNs
█ prod NaNs

metamaskbot · 2021-02-08T08:49:27Z

Builds ready [cf71e92]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
code coverage: Report
all artifacts

Page Load Metrics (607 ± 24 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	49	80	61	9	4
		domContentLoaded	533	768	605	50	24
		load	534	773	607	51	24
		domInteractive	533	768	604	50	24

kumavis · 2021-02-08T09:01:39Z

kumavis · 2021-02-08T09:52:30Z

just did a manual squash + rebase

metamaskbot · 2021-02-08T10:07:59Z

Builds ready [92afbc1]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
code coverage: Report
storybook: Storybook
all artifacts

Page Load Metrics (531 ± 33 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	44	79	56	10	5
		domContentLoaded	377	618	530	70	33
		load	379	619	531	70	33
		domInteractive	377	618	529	70	33

metamaskbot · 2021-02-08T10:35:15Z

Builds ready [92afbc1]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
code coverage: Report
storybook: Storybook
all artifacts

Page Load Metrics (1280 ± 45 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	102	148	124	14	7
		domContentLoaded	1085	1467	1276	94	45
		load	1088	1468	1280	93	45
		domInteractive	1084	1467	1276	94	45

metamaskbot · 2021-02-10T15:35:19Z

Builds ready [115911a]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
code coverage: Report
storybook: Storybook
all artifacts

Page Load Metrics (656 ± 49 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	47	88	63	11	5
		domContentLoaded	559	988	655	103	49
		load	561	989	656	103	49
		domInteractive	559	987	655	103	49

metamaskbot · 2021-02-16T04:54:08Z

Builds ready [5f5e489]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
code coverage: Report
storybook: Storybook
all artifacts

Page Load Metrics (666 ± 45 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	50	87	66	10	5
		domContentLoaded	433	842	664	93	45
		load	434	844	666	93	45
		domInteractive	432	842	664	93	45

Gudahtt

I tried testing this by deleting the readable-stream patch, updating the graceful-fs patch, and updating the policy file by running yarn lavamoat --autopolicy ./development/build/index.js prod. There are lots of new console warnings now during the build 🤔

Lots of new Buffer constructor deprecation warnings:

[DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

and many from LavaMoat about attempts to write read-access globals.

The Buffer warnings appeared to be from packages that we were using before, so I'm unsure why they didn't show up before 🤔.

I'm guessing the approach taken with the LavaMoat warnings is to assume the write attempts weren't all that important? I'm unsure how to investigate further, as the warning doesn't indicate where the write attempt occurred.

kumavis · 2021-02-18T03:55:17Z

should review console warnings

Co-authored-by: kumavis <kumavis@users.noreply.github.com>

metamaskbot · 2021-02-18T05:25:26Z

Builds ready [d526d3f]

builds: chrome, firefox, opera
bundle viz: background, ui, inpage, contentscript, ui-libs, bg-libs, phishing-detect
build viz: Build System
code coverage: Report
storybook: Storybook
all artifacts

Page Load Metrics (559 ± 8 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	43	77	54	8	4
		domContentLoaded	532	584	558	16	8
		load	534	585	559	16	8
		domInteractive	532	583	558	16	8

Gudahtt · 2021-02-18T13:01:44Z

I looked into the that "ignoring write attempt" warning, and changed it to throw an error locally to help me debug it. It turns out that the first warning I hit in any process is caused by dart. I encounter that in every process, even those that don't touch dart at all (or so I thought).

e.g.

$ yarn lavamoat ./development/build/index.js build scripts:core:dev:disable-console
yarn run v1.22.10
$ /home/gudahtt/projects/metamask-extension/node_modules/.bin/lavamoat ./development/build/index.js build scripts:core:dev:disable-console
(Error#1)

  Error#1: LavaMoat: ignoring write attempt to read-access global "process"
     at Object.set (LavaMoat/core/kernel:258:17)
    at Object.eval (/home/gudahtt/projects/metamask-extension/node_modules/sass/sass.dart.js:26:16)
    at internalRequire (LavaMoat/core/kernel:353:25)
    at requireRelative (LavaMoat/core/kernel:394:27)
    at requireRelativeWithContext (LavaMoat/core/kernel:366:16)
    at Object.eval (/home/gudahtt/projects/metamask-extension/development/build/sass-compiler.js:2:14)
    at internalRequire (LavaMoat/core/kernel:353:25)
    at requireRelative (LavaMoat/core/kernel:394:27)
    at requireRelativeWithContext (LavaMoat/core/kernel:366:16)
    at Object.eval (/home/gudahtt/projects/metamask-extension/development/build/styles.js:16:17)

Gudahtt · 2021-02-18T13:07:31Z

I was able to remove warnings from all build steps apart from styles by making sass and sass-compiler a dynamic require step. Good to know that all of those warnings were just dart.

Gudahtt · 2021-02-18T13:24:04Z

I'm seeing a bunch of changes after running yarn lavamoat:auto 🤔
I pushed them here so you could see them: db25ff6

kumavis · 2021-02-19T00:15:39Z

@Gudahtt the auto policy charges are likely because you made the require dynamic? we dont know to walk it then

Gudahtt · 2021-02-19T00:16:53Z

Sorry I meant from a clean working tree, after discarding the dynamic import change.

I figured it was due to some platform difference (mac vs Linux)

kumavis · 2021-02-19T00:33:03Z

@Gudahtt ah yeah may be optional deps diff due to platform. @EtDu is running mac and gets fsevents

Gudahtt

I was able to confirm that the policy differences were due to platform; the changes were all related to fsevents.

These cross-platform issues do present challenges to keeping the auto policy up-to-date. We'll have to postpone adding a check in CI to ensure it has been updated, as it'll always fail for contributors using macOS (and possibly Windows). But we can make do for the time being by checking prior to each release to ensure the policy file is up-to-date. I can create a CI check in a separate PR to run on release branches only, to ensure we don't accidentally forget this step.

Similarly, we can address the various console warnings in a separate PR.

LGTM!

EtDu requested review from a team and kumavis as code owners November 24, 2020 07:08

EtDu requested a review from darkwing November 24, 2020 07:08

EtDu mentioned this pull request Nov 24, 2020

Add Lavamoat to build system #9618

Closed

EtDu force-pushed the lavamoat-buildsystem branch from e9d9340 to d6aacf5 Compare November 24, 2020 07:13

EtDu force-pushed the lavamoat-buildsystem branch from d6aacf5 to 60da67e Compare December 1, 2020 02:01

EtDu added the DO-NOT-MERGE Pull requests that should not be merged label Dec 1, 2020

EtDu force-pushed the lavamoat-buildsystem branch from 111772b to 9cdd7ff Compare December 4, 2020 04:32

brad-decker assigned kumavis Dec 10, 2020

kumavis removed the DO-NOT-MERGE Pull requests that should not be merged label Dec 17, 2020

kumavis force-pushed the lavamoat-buildsystem branch from dcc16e5 to fd144d0 Compare January 28, 2021 02:08

kumavis dismissed a stale review via 56f8429 February 6, 2021 12:33

kumavis force-pushed the lavamoat-buildsystem branch from cf71e92 to 92afbc1 Compare February 8, 2021 09:50

kumavis force-pushed the lavamoat-buildsystem branch from 11a0576 to 92afbc1 Compare February 8, 2021 10:16

kumavis reviewed Feb 16, 2021

View reviewed changes

Comment thread .prettierignore Outdated

Gudahtt reviewed Feb 16, 2021

View reviewed changes

Comment thread patches/graceful-fs+4.2.3.patch Outdated

Comment thread patches/graceful-fs+4.2.3.patch Outdated

Comment thread patches/readable-stream+2.3.6.patch Outdated

Gudahtt reviewed Feb 17, 2021

View reviewed changes

Comment thread patches/stylelint+13.6.1.patch

Comment thread patches/plugin-error+1.0.1.patch

Comment thread patches/gulp-livereload++chalk+2.4.2.patch Outdated

kumavis and others added 11 commits February 18, 2021 12:34

lavamoat - run build system in lavamoat

3eaff90

lavamoat/allow-scripts - add missing policy entry

af0d151

update viz and lavvamoat

2bb5f4a

trim policy file

4f5040e

bump viz

15e9fa3

prue policy override

139e805

regen policy file

81b6719

Update package.json

2c91f56

Update package.json

4de22c5

Apply suggestions from code review

51ec72d

Co-authored-by: kumavis <kumavis@users.noreply.github.com>

update policy, remove redundant patches

8fc7b20

EtDu force-pushed the lavamoat-buildsystem branch from 5f5e489 to 8fc7b20 Compare February 18, 2021 04:59

use yarn setup in CI

d526d3f

Gudahtt approved these changes Feb 19, 2021

View reviewed changes

kumavis merged commit f196c9f into develop Feb 22, 2021

kumavis deleted the lavamoat-buildsystem branch February 22, 2021 14:43

github-actions Bot locked and limited conversation to collaborators Feb 22, 2021

Conversation

EtDu commented Nov 24, 2020 • edited by kumavis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 24, 2020

Uh oh!

metamaskbot commented Nov 24, 2020

Uh oh!

metamaskbot commented Dec 1, 2020

Uh oh!

metamaskbot commented Dec 7, 2020

Uh oh!

metamaskbot commented Dec 17, 2020

Uh oh!

metamaskbot commented Dec 17, 2020

Uh oh!

metamaskbot commented Dec 17, 2020

Uh oh!

kumavis commented Dec 17, 2020

Uh oh!

kumavis commented Dec 17, 2020

Uh oh!

Gudahtt commented Jan 23, 2021

Uh oh!

kumavis commented Feb 8, 2021

Uh oh!

metamaskbot commented Feb 8, 2021

Uh oh!

kumavis commented Feb 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumavis commented Feb 8, 2021

Uh oh!

metamaskbot commented Feb 8, 2021

Uh oh!

metamaskbot commented Feb 8, 2021

Uh oh!

metamaskbot commented Feb 10, 2021

Uh oh!

Uh oh!

metamaskbot commented Feb 16, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gudahtt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kumavis commented Feb 18, 2021

Uh oh!

metamaskbot commented Feb 18, 2021

Uh oh!

Gudahtt commented Feb 18, 2021

Uh oh!

Gudahtt commented Feb 18, 2021

Uh oh!

Gudahtt commented Feb 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumavis commented Feb 19, 2021

Uh oh!

Gudahtt commented Feb 19, 2021

Uh oh!

kumavis commented Feb 19, 2021

Uh oh!

Gudahtt left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

EtDu commented Nov 24, 2020 •

edited by kumavis

Loading

kumavis commented Feb 8, 2021 •

edited

Loading

Gudahtt commented Feb 18, 2021 •

edited

Loading