Compiler allocation accounting
The Rust implementation, pinned to de1b6c9e. Every builtin signature.
Cold require compilation uses the receiving invocation’s memory and work budgets, cancellation token and deadline. Host-side Engine::compile has no invocation budget. Both paths keep the source and syntax limits. This audit covers the current path from required-source validation through lexing, parsing, bytecode generation, diagnostics and construction of Code, including the constructors called by the generator.
The memory contract distinguishes compiler working storage from compiled metadata. Working storage remains charged while it is live. Compiled metadata, including metadata under construction, remains outside the invocation quota; source-size and cache bounds still apply. This is allocation-capacity accounting, not an RSS limit. Stack space, allocator bookkeeping, trusted host allocations and terminal runtime diagnostic formatting keep their existing exclusions. Filesystem resolution, execution, the static checker and platform support have separate requirements.
Allocation ownership
| Route | Ownership and accounting |
|---|---|
| Required-source validation and inherited hosts | Source contents use the loader’s accounted byte storage. Invalid UTF-8 creates an accounted syntax error. Module compilation borrows the receiving code’s host-name and registration pairs; the iterator creates no temporary registry or name list. Names move into Program.hosts, and matching callback descriptors are cloned into Code.hosts only after generation succeeds. |
| Lexer and editable tokens | Identifier spellings borrow the source. Token vectors, both sides of the editable token buffer, nested interpolation streams, copied streams, literal bytes, numeric text and boxed percent-word data use compiler Buffer, Bytes, Text and Boxed storage. Deferred errors share accounted text. Escaping and numeric normalization write into accounted buffers. |
| Parser, declarations and aliases | Expressions, statements, arguments, parameters, rescue clauses, class/module declarations and nested type descriptions use accounted containers. Immutable names and literal payloads share their reservations; alias copies reserve their own containers. Parser bindings, duplicate checks, visibility directives and copied scopes use accounted tables. |
| Generator working storage | Binding, parameter, read and assignment tables, copied outer scopes, declaration contexts, loop bindings, assignment discovery, temporary name lists, case patches and type-label builders remain charged. Internal slot numbers and integer decimal conversion use stack buffers. Namespace initialization traverses existing definitions directly. |
| Diagnostics | Error messages, diagnostic headers, retained filename bytes and source excerpts reserve storage before construction. Formatting counts and writes through fallible, bounded operations. Rescue handling takes over the retained allocation charge without counting a second copy. Fixed latched termination errors remain reportable after exhaustion. |
| Final program and functions | Program maps, declarations, member/global/type lists, constants, bytecode, instruction locations, parameter descriptors, local names, capture descriptors and handler lists move into compiled output. These are excluded compiled metadata. Conversion from compiler storage releases the invocation charge without keeping the originating budget alive. |
| Enums, namespaces and type literals | Enum definitions retain member spellings, normalized symbols and lookup indices; only the duplicate-check table and its keys are working storage. Namespace method/nesting vectors move directly into definitions. Runtime type descriptions, shape labels and their immutable headers become compiled metadata. Borrowed type formatting and builtin type-name classification create no intermediate type or folded-name allocation. |
| Builtin namespaces and constants | Namespace builders construct their final (Value, Value) entries and key bytes directly, sort the entries without auxiliary heap storage, and transfer them to the compiled hash. They avoid the general host-hash constructor’s staging lists and duplicate-key map. Literal constants share backing bytes through uncharged compiled wrappers; syntax owners retain their own charges until released. Regex programs and arbitrary-precision integer values are constructed by runtime instructions; the compiler validates and retains their literal text. |
| Source and exports | The source text, sparse position index, filenames, exported names and code ownership headers remain compiled metadata. Export sorting operates on the final vector. The module cache receives code only after compilation and the loader’s final interruption check succeed. |
The parser, generator and alias copies run recursive steps as tasks on a heap stack, so nesting up to the syntax depth limit needs no native stack. Like the native stack it replaces, that task stack is bounded by the depth limit and excluded from budgets; the syntax it builds stays charged.
Buffer reserves requested capacity before allocation and accounts any extra reported capacity before use. Growth holds old and new reservations together. Consuming iterators keep their reservation until the backing allocation is released. Table growth and compaction publish only after the replacement succeeds. Boxed keeps its reservation beside the value, preserving recursive parser frame sizes. Names reserve their header and bytes before copying. Source-level staging storage is not exempt merely because it feeds compiled output.
Allocation-bearing compiler helpers were traced through compilation, syntax, bytecode, code, source, enum and namespace definition builders, builtin namespace construction, shape/type formatting and literal byte storage. Direct metadata construction and its standard collection storage stay within the compiled-metadata exclusion. This audit does not change that boundary into a total-process memory guarantee.
Verification
The compiler storage and generation tests exercise exact and insufficient memory/work budgets, interruption during partial construction, aliases and copied scopes, table growth/compaction, default-stack nesting, and reclamation after failure. Diagnostic tests interrupt both formatting passes and check independently expected Unicode positions and escaped filenames. The cache lifetime test retains compiled output after dropping the first invocation and verifies that its budget is released.
Host-registration regressions check names and callback identities in a non-sorted input order, cancellation and deadlines during both metadata passes, release of partially copied callbacks, and compiled output outliving the compiling context. Nested required files exercise ordinary callbacks and block-capable methods through cold and cached loads while preserving each script’s original registrations. Invalid UTF-8 is checked for accounted rejection, exact quotas and successful retry after replacement. Builtin namespace regressions check every member name, ordering, builtin identity, numeric constant, object flag and depth.
These checks support the compiler accounting boundary.