Skip to content

[fork-ffi] numbers: toPrecisionNative — Uses string.format("%.Nf") which fixes N digits AFTER the decimal… #92

@Unisay

Description

@Unisay

Package: purescript-lua-numbers
File: src/Data/Number/Format.lua
Function: toPrecisionNative
Class: format-specifier Severity: high

Uses string.format("%.Nf") which fixes N digits AFTER the decimal point, but JS Number.prototype.toPrecision(N) produces N SIGNIFICANT digits. This is wrong for essentially every input. The module's own docstring (Format.purs lines 6-8) says toStringWith (precision 6) 1234.56789 = "1234.57" (6 sig figs), but the Lua yields "1234.567890" (6 decimal places). For small magnitudes it destroys the value: precision 3 of 0.000123456 yields "0.000" in Lua vs JS "0.000123". precision 2 of 1234.56789 should switch to exponent form "1.2e+3" in JS but Lua gives "1234.57".

Current (Lua):

toPrecisionNative = (function(d) return function(n) return string.format("%." .. tostring(d) .. "f", n) end end)

Expected: JS toPrecision(d) = d significant digits, switching to exponential notation when the magnitude is too large/small. precision 6 of 1234.56789 => "1234.57"; precision 3 of 0.000123456 => "0.000123"; precision 2 of 1234.56789 => "1.2e+3".

Proposed fix:

Use the significant-digit conversion specifier %g instead of %f: string.format("%." .. tostring(d) .. "g", n). %.6g of 1234.56789 gives "1234.57", %.3g of 0.000123456 gives "0.000123", matching toPrecision closely. (Two residual minor divergences remain vs JS: %g zero-pads the exponente+03 vs e+3and strips trailing zeros that toPrecision keeps; correcting those needs post-processing, but %g is vastly closer than %f and fixes the value-loss.)

Found by the FFI audit; reproduced under Lua 5.1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions