I had been envisioning something more like this:

try {
    return method.apply(this);
} catch (err) {
    if (err instanceof assert.AssertionError) {
        throw Object.assign({}, new assert.AssertionError, { message: `${e.actual} ${e.operator} ${e.expected}` })
    }
    throw err;
}

That way, any unrelated errors (e.g. crashes in a rule) would still get passed through unmodified, and the original message (e.g. "output is incorrect") would still be present for additional context.

@not-an-aardvark using a plain object as error object is not the same as throwing a new AssertionError. Using Object.assign is also not sufficient because stack and message are non-enumerable and would not be copied.

Oops, you're right; I had forgotten that the thrown object would no longer be an AssertionError. I think something like this could work:

try {
    return method.apply(this);
} catch (err) {
    if (err instanceof assert.AssertionError) {
        err.message += `${util.inspect(err.actual)} ${err.operator} ${util.inspect(err.expected)}`;
    }
    throw err;
}

To me, the important constraints are:

• We don't add redundant info in the case where a test runner is being used (this is by far the most common case)
• The error message still contains information about what the problem is (e.g. it should still say something like "output is incorrect" when the output is incorrect, rather than just displaying a diff between two strings)

I'd feel better if this logic (from here to the end of the assertion) were wrapped in a try/finally so that the globals are definitely reassigned even if an exception is thrown.

I've added the try/finally : )

Can you make this append to the existing message rather than overwriting it? I think it would be good to retain the "Output is incorrect" text to make the source of the error clearer.

Oh, I forgot to explain here.
The defaultHandler is called nestedly somehow (sorry I don't dive into the code but its behaviour shows that), so the thrown error will be caught and err message
will be appended in multiple times. It will finally become like

"Output is incorrect 'a' == 'b' 'a' == 'b' 'a' == 'b'"

That's the reason for overwriting it here.

Another solution is adding a flag like appended on error object to avoid multiple appending. But is it worth to do it?

Oh, I see. I think that's happening because defaultHandler is used for both the "describe" and the "it" methods. So effectively something like this is happening:

defaultHandler("foo", () => {
    defaultHandler("bar", () => {
        // throw assertion error
    });
});

One solution would be to use the old default handler (which just calls the function in the second argument) as the default for describe, and use the handler that appends a diff as the default for it.

I think surrounding the comparison in parentheses would be useful and make the error message more readable: (${util.inspect(err.actual)} ${err.operator} ${util.inspect(err.expected)})

Thanks for review. I've added the parentheses :-)