Skip to content

NormalModule.needRebuild cache calculations seem to be wrong for filesystems that don't support millisecond precision mtimes #2003

@timmfin

Description

@timmfin

I was writing some integrations tests for broccoli-webpack-cached (trying to test that broccoli and webpack caching were working well together) and I was getting pretty unexpected, seemingly non-deterministic behavior. That lead me to digging into how Webpack caches things, namely how:

  • NormalModule saves a buildTimestamp every time a file is built
  • At the start of a new compile, Webpack gathers the mtimes from the filesystem for of every file in the dependency tree via CachePlugin
  • And then during the next compile, compares the new mtime against the previously saved buildTimestamp for each file in needsRebuilt

That all makes sense, but the problem I see is that buildTimestamp is saved with millisecond-level precision. However, many filesystems out there (in my case, HFS on OSX) only support second-level precision. So this can happen:

  • A compile starts, let's say at at 13:50:00 and 100ms (hour, min, second, and ms). That is saved to this.buildTimestamp.
  • The compile ends, the file is modified immediately at 13:50:00 and 500ms. But the timestamp saved in the compiler.fileTimestamps object has no millisecond precision, so is only 13:50:00 and 0ms
  • Another compile is kicked off soon after and checks if that file needs to be rebuilt. It compares to see if the new filesystem timestamp (13:50:00 and 0ms) is >= the previous buildTimestamp (13:50:00 and 500ms) and decides there has not been a file modification. Even though there definitely was a change to the file, it just happened within the same second and was ignored.

I verified this problem goes away when I force NormalModule to truncate milliseconds from saved the buildTimestamp, like so:

this.buildTimestamp = Math.floor(new Date().getTime() / 1000) * 1000;

But that likely isn't a good change. Ideally, I think that truncation would only happen if we can detect that the input filesystem only has second-level precision (but I didn't look deep how hard/easy that is).

I realize that I am much more likely to run into this issue when writing test code, and that is pretty different than a typical local dev server/watch/build script use-case. But I gotta think that this bug even rears its head in normal scenarios from time to time... well that is unless I've missed something.

ps: Above Webpack github links are to 1.x not 2.x-beta source.

pps: Node didn't always, but now does support millisecond precision for filesystems that support it.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions