Skip to content

feat: lift the pure subset of foreign values into IR primops (#178) - #210

Merged
Unisay merged 6 commits into
mainfrom
issue-178/foreign-lift-primops
Jul 8, 2026
Merged

feat: lift the pure subset of foreign values into IR primops (#178)#210
Unisay merged 6 commits into
mainfrom
issue-178/foreign-lift-primops

Conversation

@Unisay

@Unisay Unisay commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Closes #178. Builds on the foreign parser (#173) and case-of-known-constructor (#177); unblocks the dictionary-method inlining of #180.

What

Polymorphic hot code bottoms out in opaque curried foreigns (intAdd, ordIntImpl, refEq, boolConj) that the IR optimizer cannot see through, because to the IR a foreign body is text. This lifts the pure return-tree subset of such a body, taken from the actual fork source the foreign parser (#173) already gives us, into IR primops the existing rewrites can specialize.

  • IR primops (IR.Types): PrimBinOp ann PrimOp a b and PrimNot ann a, with a PrimOp enum of the twelve binary operators. Each node is defined as the matching Lua operator, so lowering is the identity and lifting its inverse (Note [IR primops]). Equality stays the existing Eq node; the lifter maps == onto it and ~= onto not (a == b).
  • Lowering (Lua.fromIR): each primop lowers to the same-named Lua BinOp / UnOp.
  • Constant folding (IR.Optimizer): fold primops over literal operands in Lua 5.1 semantics (Note [Folding primops follows Lua 5.1]). Integer +/-/* fold with a ±2^53 exactness guard, floats only to a finite result, integer modulo copies the divisor-signed a - floor(a/b)*b, concat is string-with-string, comparisons fold on two numbers, and and/or/not fold including the short-circuit identities.
  • Boolean-if simplification (IR.Optimizer): if p then True else False becomes p, if p then False else True becomes not p, if not p then a else b becomes if p then b else a (drop the negation, swap the branches), and not (not e) becomes e. Every Ord comparison and /= decays to the if … then False else True shape, so once the comparison bodies lift this is the dominant residual; the not p rewrite is only expressible because the primops above give the IR a PrimNot. The Lua backend then emits not (a == b) as a ~= b, the form luacheck expects.
  • The lifter (Lua.ForeignLift): translate the liftable subset of an export's parsed Lua AST. Curried single-parameter lambdas become nested Abs, if/elseif/else return trees become IfThenElse, operators become primops, and a reference resolves either to a parameter or to an inlined header local (which is how ordIntImpl = (unsafeCoerceImpl) and the refEq aliases lift). Loops, mutation, varargs, multi-parameter functions, table constructors, and calls leave the export opaque.
  • The allowlist: the arithmetic/comparison/boolean/concat core (intAdd/intMul/intSub, ordIntImpl/ordCharImpl, boolConj/boolDisj/boolNot, the refEq family, concatString). It is a hard contract: an allowlisted export that fails to lift, because a fork changed its shape, is a compile error rather than a silent performance cliff. An export not on the list stays opaque even when it would lift.
  • Wiring: Backend.compileModules runs the lifter before the optimizer, and the golden harness's compileCorefn does the same, so the .ir goldens go through the real pipeline.

Design points

Lifted exports become ordinary bindings, not foreigns. The first cut replaced the accessor in place inside uberModuleForeigns, and it miscompiled. DCE's reachability graph only understands the ForeignImport / ObjectProp shapes among the foreigns, so a lifted lambda has no vertex there, gets read as unreachable, and is dropped while every reference to it survives dangling (attempt to call field 'Data_Ring_intSub' (a nil value)). Moving the lifted export into uberModuleBindings fixes it structurally: DCE handles it as a normal binding, and once the accessor is gone it prunes the export's name from the module's ForeignImport, since nothing reads it off the foreign import anymore, so the source row drops out of the emitted FFI. When every export of a module lifts, its ForeignImport disappears: see the Fibonacci diff, where the Data_Semiring_foreign / Data_Ring_foreign tables are gone and intAdd is a native +.

Folding tracks Lua 5.1. The target is all-doubles, so the rules watch it: integer folds bail beyond ±2^53 (a folded literal has to reparse to the value the runtime computes), float folds stop at a finite result (Lua has no inf/nan numeric literal), modulo copies the formula whose sign follows the divisor, and concat is string-with-string only (Lua's number-to-string on concat is build-dependent). Comparisons fold on two numbers; strings and chars are left alone, since Lua orders strings by bytes while the IR literal is semantic Text.

Semantics from the source, not a registry. The original plan was a hand-written qualified-name to IR registry (the purs-backend-es approach). Deriving the semantics from the parsed fork source instead makes registry drift against the package set impossible by construction. That is the payoff of Lua's tiny grammar, and the reason the allowlist can be a hard contract instead of best-effort.

Verification

  • Unit tests for every folding rule (the range guard, the modulo sign, finite-float, string concat, numeric comparisons, boolean short-circuit) and its declines, for the boolean-if rules and the ~= peephole, and for the lifter (each liftable shape, each declined shape, allowlist membership).
  • The shared IR generator now emits PrimBinOp / PrimNot, so the optimizer's well-scopedness, GUC, free-reference, and idempotence properties fuzz constant folding over primops. Unlike Fold ReflectCtor and DataArgumentByIndex over known constructor applications (case-of-known-constructor) #177, widening the generator is in scope here, since folding correctness is what benefits from the fuzz.
  • A runnable Golden.Primops module exercises lifted arithmetic, comparison, equality, concat, and booleans, with a hand-verified eval oracle.
  • Full suite green (686 examples), hlint clean, warning-free build. Every eval oracle is unchanged across the golden churn, which is the semantic safety net: the runtime output of every runnable golden is byte-identical, so both the lifting and the boolean-if simplification are behavior-preserving. The boolean-if rules also shrink the output, e.g. the tailRecM loop guard collapses from a nested IIFE plus a boolean round-trip to a single swapped-branch if.

Notes for reviewers

Scope vs the 42x number. #178 gives you the primops and the lifter, and direct arithmetic collapses fully (the Fibonacci golden is native + and subtraction now, no foreign dispatch). The general greaterThanOrEq(ordInt)(a)(b) to not (a < b) collapse also needs dictionary-method inlining (#180), which this unblocks: the ordIntImpl body is already lifted into primops inside ordInt.compare, but the Data_Ord_compare(dict) method dispatch stands until #180 inlines it.

Twice-used primops can surface as a shared worker. A lifted primop used at two or more saturated sites can come out as an uncurried worker instead of an inline operator (Fibonacci's sub$w(v, k) for n - 1 and n - 2). The chain: purs shares sub ringInt as a top-level binding when - is used more than once, reduceObjectProp strips the Just Always off the lifted lambda as it resolves through the dict projection (by design, so foreign accessors don't over-inline), and uncurry then splits the surviving curried binding into worker and wrapper. Correct, and still far cheaper than the old foreign dispatch of a table index plus two curried calls. Inlining trivial primop-bodied bindings regardless of use count would close the gap; it is a reasonable low-priority follow-up, filed as #211 and out of scope here.

Unisay added 2 commits July 8, 2026 13:26
Polymorphic hot code bottoms out in opaque curried foreigns — intAdd,
ordIntImpl, refEq, boolConj — that the IR optimizer cannot see through,
because to the IR a foreign body is text. This lifts the pure return-tree
subset of such a body, taken from the actual fork source parsed by the
Lua foreign parser (#173), into IR primops that the existing rewrites
(beta reduction, case-of-known-constructor #177, inlining, constant
folding) can then specialize.

- IR.Types: PrimBinOp / PrimNot nodes and the PrimOp enum, defined as the
  matching Lua operators so lowering is the identity and lifting its
  inverse (Note [IR primops]). Equality stays the existing Eq node.
- Lua.fromIR: lower each primop onto the same-named Lua BinOp / UnOp.
- IR.Optimizer: constant-fold primops over literals following Lua 5.1
  semantics (integer ±2^53 guard, finite-only floats, divisor-signed
  modulo, string-only concat, numeric comparisons, boolean short-circuit).
- Lua.ForeignLift: translate the liftable subset (curried single-parameter
  lambdas, if/elseif/else return trees, operators, header-local inlining)
  to IR, gated by a hard allowlist — a listed export that fails to lift is
  a compile error, never a silent performance cliff. Lifted exports move
  out of the foreigns into ordinary bindings, so DCE prunes the now-unread
  source rows from the emitted FFI.
- Backend.compileModules runs the lifter before the optimizer.
- IR.Optimizer.Spec: unit tests for every folding rule (integer range
  guard, Lua-5.1 modulo sign, finite-float, string concat, numeric
  comparisons, boolean short-circuit) and the non-folding cases.
- Lua.ForeignLift.Spec: the liftable shapes (curried operator, header-local
  alias, if/elseif/else through a header local, not, concat) and the
  declined ones (multi-parameter, table index, non-return-tree, else-less
  if, missing export), plus allowlist membership.
- IR.Gen: emit PrimBinOp / PrimNot in both generators, so the optimizer
  soundness properties fuzz folding for scope/GUC preservation.
- Golden.Primops: a runnable module exercising lifted arithmetic,
  comparison, equality, concat and booleans, with a hand-verified eval
  oracle.
- Golden harness: lift foreigns in compileCorefn too, so the .ir goldens
  reflect the same pipeline; regenerate the structural goldens (every eval
  oracle is unchanged — semantics preserved).
@Unisay
Unisay requested a review from Copilot July 8, 2026 11:40
@Unisay Unisay self-assigned this Jul 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the PureScript→Lua compiler pipeline by lifting a strictly pure subset of allowlisted foreign exports into new IR primop nodes, enabling downstream IR optimizations (including constant folding) to “see through” previously-opaque foreign definitions and generate more direct Lua operator code.

Changes:

  • Introduces IR primop nodes (PrimBinOp + PrimOp enum, and PrimNot) and lowers them directly to Lua operators.
  • Adds Lua foreign lifting pass (allowlist-driven) and wires it into both Backend.compileModules and the golden-test pipeline.
  • Implements Lua-5.1-semantics constant folding for primops, expands IR fuzz generators accordingly, and updates/adds tests and golden fixtures (including a new runnable Golden.Primops).

Reviewed changes

Copilot reviewed 65 out of 74 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
test/ps/src/Golden/Primops/Test.purs New runnable golden exercising lifted primops via user-level PureScript code.
test/ps/output/Golden.Primops.Test/golden.lua New golden Lua output for Golden.Primops.
test/ps/output/Golden.Primops.Test/eval/golden.txt New eval oracle (hand-verified stdout) for Golden.Primops.
test/ps/output/Golden.Primops.Test/eval/.gitignore Tracks eval harness output file.
test/ps/output/Golden.Uncurry.Test/golden.lua Golden updated to reflect lifted foreigns/primops reducing foreign dispatch.
test/ps/output/Golden.TailRecM2Shadow.Test/golden.lua Golden updated to reflect lifted ordIntImpl and arithmetic primops.
test/ps/output/Golden.TailRecM2Shadow.Test/golden.ir Golden IR updated to include primop nodes and lifted bindings.
test/ps/output/Golden.StringCodePoints.Test/golden.lua Golden updated for lifted Eq/Ord/Semiring/Semigroup/HeytingAlgebra foreigns.
test/ps/output/Golden.ProfunctorDictLens.Test/golden.lua Golden updated to show direct operator lowering rather than foreign calls.
test/ps/output/Golden.ProfunctorDictLens.Test/golden.ir Golden IR updated to include primops in dictionaries and call sites.
test/ps/output/Golden.MaybeChainModule.Test/golden.lua Golden refreshed due to upstream optimizer/pipeline changes.
test/ps/output/Golden.MaybeChainModule.Test/golden.ir Golden IR refreshed due to upstream optimizer/pipeline changes.
test/ps/output/Golden.MaybeChain.Test/golden.lua Golden refreshed due to upstream optimizer/pipeline changes.
test/ps/output/Golden.MaybeChain.Test/golden.ir Golden IR refreshed due to upstream optimizer/pipeline changes.
test/ps/output/Golden.Loopification.Test/golden.lua Golden updated for lifted foreigns and primop lowering (e.g. subtraction worker).
test/ps/output/Golden.Loopification.Test/golden.ir Golden IR updated to reflect new primop nodes and lifted bindings.
test/ps/output/Golden.LongWriterBind.Test/golden.lua Golden refreshed due to upstream optimizer/pipeline changes.
test/ps/output/Golden.LongWriterBind.Test/golden.ir Golden IR refreshed due to upstream optimizer/pipeline changes.
test/ps/output/Golden.LongReaderBind.Test/golden.lua Golden updated to reflect lifted arithmetic foreigns into primops/workers.
test/ps/output/Golden.LongReaderBind.Test/golden.ir Golden IR updated to include primops/lifted bindings.
test/ps/output/Golden.LongCallbackChain.Test/golden.lua Golden updated to reflect lifted ordIntImpl and arithmetic primops.
test/ps/output/Golden.LongCallbackChain.Test/golden.ir Golden IR updated to include primops/lifted bindings.
test/ps/output/Golden.LongBindFlipped.Test/golden.ir Golden IR updated to include string concat primop and arithmetic primops.
test/ps/output/Golden.LongApplyChain.Test/golden.ir Golden IR updated to include string concat primop and refreshed temps.
test/ps/output/Golden.Issue37.Test/golden.lua Golden refreshed (identifier renames/optimizer churn).
test/ps/output/Golden.Issue37.Test/golden.ir Golden IR refreshed (identifier renames/optimizer churn).
test/ps/output/Golden.GenericEqTwoTypes.Test/golden.lua Golden updated to reflect lifted boolean/equality foreigns.
test/ps/output/Golden.GenericEqTwoTypes.Test/golden.ir Golden IR updated to include primops for boolean ops and lifted Eq.
test/ps/output/Golden.FloatIn.Test/golden.lua Golden updated to remove foreign dispatch for basic arithmetic.
test/ps/output/Golden.FloatIn.Test/golden.ir Golden IR updated to include primops in Semiring dictionaries.
test/ps/output/Golden.FieldCaching.Test/golden.lua Golden updated for lifted Eq/Ord/Ring/Semiring foreigns and primop lowering.
test/ps/output/Golden.FieldCaching.Test/golden.ir Golden IR updated to include primops/lifted bindings and simplified Eq check.
test/ps/output/Golden.Fibonacci.Test/golden.lua Golden updated to use direct + and a lifted subtraction worker.
test/ps/output/Golden.Fibonacci.Test/golden.ir Golden IR updated to include primops and lifted sub$w.
test/ps/output/Golden.DerivedFunctor.Test/golden.lua Golden updated to remove foreign dispatch for arithmetic.
test/ps/output/Golden.DerivedFunctor.Test/golden.ir Golden IR updated to include primops in Semiring dictionaries.
test/ps/output/Golden.CharLiterals.Test/golden.lua Golden updated for lifted Eq/Ord foreigns (direct operators in compare/equality).
test/ps/output/Golden.CharLiterals.Test/golden.ir Golden IR updated to include lifted comparisons and simplified equality.
test/ps/output/Golden.BugListGenericEq.Test/golden.lua Golden updated to reflect lifted boolean/equality foreigns.
test/ps/output/Golden.BugListGenericEq.Test/golden.ir Golden IR updated to include primops for boolean ops and lifted Eq.
test/ps/output/Golden.ArrayPatternMatch.Test/golden.lua Golden updated to remove foreign dispatch for arithmetic dictionaries.
test/ps/output/Golden.ArrayPatternMatch.Test/golden.ir Golden IR updated to include primops in Semiring/Ring dictionaries.
test/ps/output/Golden.ArrayOfUnits.Test/golden.lua Golden updated to remove foreign dispatch for arithmetic dictionaries.
test/ps/output/Golden.ArrayOfUnits.Test/golden.ir Golden IR updated to include primops and refreshed identifiers.
test/Main.hs Registers the new ForeignLift test suite.
test/Language/PureScript/Backend/Lua/Golden/Spec.hs Wires foreign lifting into the golden IR pipeline (mirrors compileModules).
test/Language/PureScript/Backend/Lua/ForeignLift/Spec.hs New unit tests for lifting behavior and allowlist membership.
test/Language/PureScript/Backend/IR/Optimizer/Spec.hs Adds unit tests for primop constant folding and its decline cases.
test/Language/PureScript/Backend/IR/Gen.hs Extends IR generators to emit primops for property-based fuzzing.
pslua.cabal Adds the new Lua.ForeignLift module to library and test-suite stanzas.
lib/Language/PureScript/Backend/Lua.hs Lowers IR primops directly to Lua BinOp/UnOp operators.
lib/Language/PureScript/Backend/IR/Types.hs Adds primop AST nodes + helpers, and integrates them into traversal/alphaEq.
lib/Language/PureScript/Backend/IR/Optimizer.hs Implements Lua 5.1–semantics constant folding for primops.
lib/Language/PureScript/Backend.hs Wires ForeignLift.liftForeigns into the main compilation pipeline.
exe/Main.hs Adds ForeignLift error handling to the CLI entrypoint.

Comment thread test/ps/src/Golden/Primops/Test.purs Outdated
Every `Ord` comparison and `/=` decays to `case … of Ctor -> false; _ ->
true`, which compiles to `if p then False else True`; #178's lifted
comparison bodies made this the dominant residual shape (spotted in
review). The IR optimizer now folds it:

- `if p then True else False` -> `p`
- `if p then False else True` -> `not p` (a `PrimNot`, a node the IR only
  gained with #178's primops)
- `if not p then a else b` -> `if p then b else a` (drop the negation and
  swap the branches; runs before the two literal rules so `if not p then
  False else True` normalises to `p` rather than stalling at `not (not p)`)
- `not (not e)` -> `e`

The Lua backend additionally emits `not (a == b)` as `a ~= b` (Lua's `~=`
is exactly the negation of `==`), the form luacheck expects.

Together these collapse e.g. the tailRecM loop guard from a nested IIFE
plus a boolean round-trip to a single swapped-branch if (net -105 lines
across 16 goldens). Every eval oracle is unchanged.
@Unisay
Unisay marked this pull request as ready for review July 8, 2026 13:57
Unisay added 3 commits July 8, 2026 16:01
- test/ps/src/Golden/Primops/Test.purs:9: reword the comment — arithmetic
  lifts to direct operators, while `n <= 0` still routes through the lifted
  compare (dispatch collapse is #180)
  (#210 (comment))
The Copilot-review reword in 2d1031c added two comment lines to
Golden/Primops/Test.purs but did not regenerate the committed CoreFn, so
every sourceSpan in corefn.json was two lines stale. Re-emit it from the
current source. The structural goldens (golden.ir / golden.lua) carry no
source line numbers, so they are unaffected.
The #178 codegen change moved Bench.BindChain's inner continuation closure
onto LuaJIT's hot-count boundary: its function entry root-traces in roughly
one process out of five and stays interpreted in the rest, a coin flip
weighted by LuaJIT's entropy-seeded penalty PRNG. bench/ci generated each
trace report twice and compared byte for byte, so that marginal spot made
the two runs differ about a third of the time and flaked CI.

trace_report.lua now runs the workload in N independent luajit subprocesses
(default 9, BENCH_TRACE_TRIALS overrides) and reports only the abort sites
and end states that every trial agrees on. Decisive spots (hot loops,
blacklisted entries, reliably root-traced workers) are stable across every
run, so the intersection keeps them; the boundary noise drops out. The
canonical report is stable by construction, so bench/ci runs each trace
report once instead of twice-and-compare. The FNEW census is a pure static
function of the artifact, so it keeps its twice-and-compare byte-stability
proof.

Accept the bench goldens moved by #178. The headline wins: Bench.Fib and
Bench.CurriedStep lose every "NYI: bytecode FNEW" trace abort (arithmetic is
native primops now, no per-call closures) and neither blacklists anything.
The FNEW census drops accordingly: Fib function-body FNEW 2 to 0, CurriedStep
total FNEW 9 to 2.
@Unisay
Unisay merged commit ef55831 into main Jul 8, 2026
2 checks passed
@Unisay
Unisay deleted the issue-178/foreign-lift-primops branch July 8, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lift the pure subset of foreign values into the IR: primop nodes and an allowlist-driven lifter

2 participants