Skip to content

refactor: use flush option with writeFile#5058

Merged
B4nan merged 2 commits intomikro-orm:masterfrom
boenrobot:writeFileFix
Dec 29, 2023
Merged

refactor: use flush option with writeFile#5058
B4nan merged 2 commits intomikro-orm:masterfrom
boenrobot:writeFileFix

Conversation

@boenrobot
Copy link
Copy Markdown
Collaborator

In all places of the code, fs-extra's writeFile is used. This package however is actually a re-export of the callback version of node:fs' writeFile.
Meanwhile, all packages seem to assume this is the version from node:fs/promises.

This fix in turn fixes tests that can sometimes be flaky, due to assuming file operations are done, when that isn't necessarily true.

Also reordered imports in the affected files alphabetically.


I noticed this while running tests locally. Random tests sometimes fail, and all they had in common was the fact that they were checking for a file's existence. I thought it's maybe an fs-extra specific issue with Windows, but then discovered it's not re-exporting the promise, hence this PR.

@codecov
Copy link
Copy Markdown

codecov bot commented Dec 28, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (b523ae4) 99.78% compared to head (0100ed9) 99.78%.
Report is 3 commits behind head on master.

❗ Current head 0100ed9 differs from pull request most recent head 72007ec. Consider uploading reports for the commit 72007ec to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5058   +/-   ##
=======================================
  Coverage   99.78%   99.78%           
=======================================
  Files         221      221           
  Lines       16345    16351    +6     
  Branches     3933     3936    +3     
=======================================
+ Hits        16310    16316    +6     
  Misses         35       35           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@B4nan
Copy link
Copy Markdown
Member

B4nan commented Dec 28, 2023

This package however is actually a re-export of the callback version of node:fs' writeFile.

Are you sure about that? The library itself is using it by awaiting internally:

https://github.com/jprichardson/node-fs-extra/blob/master/lib/ensure/file.js#L24

Also the types say it's the promisified version.

If it's true, it sounds like something worth reporting and fixing upstream.

FWIW, I don't think I saw a flaky test that would point to this problem, so could be surely windows specific.

@B4nan
Copy link
Copy Markdown
Member

B4nan commented Dec 28, 2023

I don't think this is true, when I console.log the method without awaiting, I get a Promise instance back.

import { writeFile } from 'fs-extra';

console.log(writeFile('foo', 'bar'));

Promise { <pending> }

Added "flush: true" to ensure that all files written are immediately accessible.
The code already collects the entire contents of a file to write it in one go,
and promises are used to prevent blocking the event loop.

This in turn fixes potentially flaky tests, though the cost is that the overall
time until all file writing promises are resolved will be a bit more.

Also reordered imports in the affected files alphabetically.
@boenrobot
Copy link
Copy Markdown
Collaborator Author

boenrobot commented Dec 29, 2023

Hm... after digging through more sources, it seems the "core" of this is in graceful-fs. fs-extra is re-exporting that, and wrapping it up with universalify to make it promise based.

So... yes, there is a promise there, but it seems there may be some kind of an edge case, at least on Windows, where the native promise writeFile handles things properly, while the universalify writeFile does not.

The graceful-fs package mentions in their readme

On Windows, it retries renaming a file for up to one second if EACCESS or EPERM error occurs, likely because antivirus software has locked the directory.

which leads me to believe that may be what's happening here, except I don't get those errors... It's just that pathExists sometimes returns false, even though it shouldn't. I'm guessing I would've gotten those exact errors if the test was trying to read the file, not just check for its existence.

Or, equally likely, the native promise version is scheduling the flushes differently, and so without an explicit flush option or filehandle.sync() call, the end result is that the flush is scheduled later with the non-natively promised version.

A more reliable fix would be to take advantage of the flush option in Node 21 I guess (see nodejs/node#50009 ), which would be ignored for versions that don't support it. I've now changed the PR to that. I re-ran the tests several times, and so far at least, I haven't gotten a single flaky test.

Comment thread scripts/copy.mjs Outdated
@B4nan B4nan changed the title fix(fs): fixed usages of writeFile refactor: use flush option with writeFile Dec 29, 2023
@B4nan B4nan merged commit 0fe15fe into mikro-orm:master Dec 29, 2023
@boenrobot boenrobot deleted the writeFileFix branch February 10, 2024 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants