Replace the lexical foreign splitter with a real Lua 5.1 parser producing the Lua AST #173

Problem

Lua.Linker.Foreign does not parse foreign files — it lexically splits them into a header text and (Key, Text) export pairs, extracting each value as a raw text blob by counting balanced parentheses (hence the "every value is wrapped in parens" contract in Note [Foreign module source format]). The blobs are embedded into the Lua AST as opaque verbatim nodes, so every optimization stops at a foreign boundary.

The measurable cost: when the inliner pastes a foreign lambda into an application site, the result is an applied function literal the optimizer cannot see into. Golden.Fibonacci compiles every addition to (function(x) return function(y) return x + y end end)(a)(b) — two closure allocations and two calls per arithmetic op. fib(30) runs 8.3x slower than handwritten Lua under PUC 5.1; under LuaJIT the per-call FNEW aborts every trace recording and gets the loop blacklisted, so the JIT contributes nothing.

Approach

Implement our own Lua 5.1 parser (megaparsec is already a dependency) targeting the existing Lua.Types AST:

1. Full Lua 5.1 expression/statement grammar (it is small). Each foreign export value parses into an AST expression, the header into statements.
2. Comment preservation: comments attach to AST nodes via the existing annotation slots, so printed output keeps the load-bearing FFI comments and structural goldens stay close. Formatting/blank-line fidelity is a non-goal.
3. The balanced-paren scanner and the parens-around-values contract become unnecessary; the overall file format contract (optional header, then return { ... }) stays.
4. Foreign files get compile-time syntax validation — today a syntax error inside a value blob is caught only by luacheck or at runtime.

Decision recorded: own parser, not hackage language-lua — the values must land in our Lua.Types with comments attached, and converting from a third-party comment-less AST buys nothing. This parser is a standing decision for the series: later issues build on foreign values being real AST.

Prerequisites / Relations

Infrastructure with no hard prerequisite (megaparsec is already a dependency). What it unlocks:

• Immediately, a Lua-level beta-reduction rule for applied function literals: FunctionCall (Function ps body) args substitutes trivial arguments (variable/literal) and local-binds non-trivial ones — the same work-duplication discipline betaReduce substitutes a non-trivial argument into every occurrence of the parameter, multiplying work #167 asks of the IR-level betaReduce. This dissolves the foreign IIFE pattern above and generalizes the special-cased foreign scope-IIFE folds (IR.Abs lowering wraps a chunk body in a scope IIFE that a separate rule then strips #158, Fold a field projection through a foreign-header scope IIFE #159).
• Later, lifting the pure return-tree subset of foreign values into the IR as primitive operations derived from the actual fork sources (Lift the pure subset of foreign values into the IR: primop nodes and an allowlist-driven lifter #178); this parser is that work's prerequisite. Foreign table-constructor exports are duplicated per use site (fresh identity and allocation per copy) #175's principled "is a table constructor" check also wants this parser.

Verification / Measurement

The fib(30) macrobench and FNEW counters from #172 show the win directly: the foreign IIFE ((function(x) return function(y) return x + y end end)(a)(b), two closure allocations + two calls per add) collapses, and the per-call FNEW that aborts LuaJIT trace recording disappears. Separately, a foreign file with a syntax error inside a value now fails at compile time rather than only under luacheck or at runtime.