Skip to content

feat(codemode): add embeddable Rust port - #64437

Draft
kommander wants to merge 264 commits into
devfrom
codemode-rust
Draft

kommander wants to merge 264 commits into
devfrom
codemode-rust

Conversation

@kommander

@kommander kommander commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

None.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Staging branch for a Rust implementation of the PsychoScript language layer (packages/codemode-rust, crate psychoscript), the counterpart of packages/codemode/src/language. It is pure Rust, with no Bun or Node, and covers the language only: Code Mode (tools, search, OpenAPI) stays a TypeScript layer on top. It will move to a PR on anomalyco/psychoscript once the language split is synced there. See packages/codemode-rust/PR.md.

  • Parity in both directions. Every expectation is recorded from the TypeScript language layer, and CI regenerates and diffs the recordings. test262 is 13,444/13,444, with both gap lists empty.
  • Threading. A program compiles on the caller's thread unless its nesting needs more stack. Long compute in a run driven as a future moves to a worker, while host code stays on the executor.
  • Speed. About 1.4–4.0× faster than the TypeScript language runtime on the nine bench programs (bench/RESULTS.md).
  • packages/codemode. It gets JavaScript-correctness fixes, acorn 8.16, and the fix for a blocking call inside a sync generator.

How did you verify your code works?

The Rust checks run in check-test.yml's unit job: fmt, clippy, the workspace tests, round-trip conformance, the recordings' freshness check and test262. bun test and bun run check also pass.

Screenshots / recordings

N/A

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

A run defines search, require, the import hook and extension globals only
for the names its program refers to, and a module import makes the one it
asks for. The run id is made the first time a call's key is read.
…Primitive

Operators and built-in arguments now call their valueOf/toString like any other object, as JS does: fn + 1 is "[object Function]1" and -fn is NaN. Promises and tool references still reject, so a missing await cannot hide.
EvaluateNew evaluates the callee and its arguments before IsConstructor, so an argument's side effects and throws come first. Bumps the compiler version, since NEW no longer emits a separate check.
PutValue checks the base, so null.x = f() calls f before it throws, and (1).x = f() surfaces f's throw.
…owns the key

delete "abc"[5] and delete (1).x are true, as they delete from the wrapper; a string's own index or length still throws in strict mode.
Resolving functions are anonymous, Set.prototype.keys is values, built-in iterators report their kind (Array Iterator, Map Iterator, ...), AggregateError has no own message unless given one, and generator and async functions inherit from their own non-callable function prototypes.
The fast path for an integer in an array answered false for a hole; it now answers only for a present element and leaves the rest to the prototype walk, so forEach, some, and reduce visit an index inherited from Array.prototype.
…ough ToPrimitive

The integer arguments of the Array methods, length = object, and the elements a default sort compares call the program's own valueOf/toString, as do String.fromCharCode/fromCodePoint, JSON.parse, and the string iterator's receiver. repeat(NaN) is "", as ToIntegerOrInfinity makes NaN 0.
openapi::from_spec ports OpenAPI.fromSpec: one tool per representable
operation with dotted operationIds as namespaces, the reference's skip
reasons, readOnly/writeOnly projection solved over reference cycles, OpenAPI
3.0 and 3.1 schema normalization, host-side auth resolution, and request
building over an HttpClient trait the host implements.

conformance/catalog.ts now also runs the documents of the reference's
openapi.test.ts and its fixtures, recording signatures, schemas, skipped
operations, and each call's requests, credentials and result; the Rust test
replays them with a fake client.
A tool returning ToolError::interrupt() stops the run, as a tool interrupting
its fiber does in the reference: Program::run, run_async and execute_async
end with Interrupted instead of an outcome, calls still running are
interrupted with it, and tool.after sees the call as interrupted. The
blocking execute keeps its signature and reports an ExecutionFailure.
A rejected promise nothing handles by the end of a successful run becomes a
warning in Outcome.warnings, in promise creation order, as in the reference,
after completed host calls settle. Warnings get their own output budget and
a Truncated marker. The VM side is two hooks in promise.rs: a rejection with
no reactions, and a reaction attached to a rejected promise.
…and key order

- An OpenAPI 3.0 schema with a keyword only JSON Schema 2020-12 defines
  (const, $id, examples, if, ...) fails openapi::from_spec with the error
  the reference throws; from_spec now returns Result<Generated, SpecError>.
- The 3.0 converter records where each schema lands, so references into a
  nullable allOf that becomes anyOf follow it, as Effect's converter does.
- Specs and tool schemas take JavaScript's key order, integer-like keys
  first, so signatures and generated schemas render in the reference's
  order.

catalog.ts covers each with documents written as JSON text, and the OpenAPI
test now compares generated schemas key order included.
Regular expression literals, the RegExp constructor with flags, lastIndex,
exec/test, RegExp.escape, and match, matchAll, search, replace, replaceAll
and split with regex patterns, replacement templates and function
replacers.

Adds the regress crate (pure Rust, ECMAScript regex semantics, UTF-16
matching); the stripped embed example grows from 2139576 to 3047304 bytes.

The RegExp pattern, exec/test input and lastIndex convert without consulting program methods, as the reference's coerceToString/coerceToNumber do. test262: the files this fixes leave rust-skipped.txt, and the reference's skipped.txt drops Array lastIndexOf 15.4.4.15-5-21, which the reference passes since its Array arguments go through ToPrimitive.
normalize uses the unicode-normalization crate (pure Rust, the standard
NFC/NFD/NFKC/NFKD tables); the stripped embed example grows from 3047304
to 3178872 bytes. Program strings are UTF-8, so isWellFormed is always
true and toWellFormed returns its receiver. repeat and padStart report the
reference's limit errors.
The Date constructor in every form, Date.now/parse/UTC, the local and UTC
getters and setters with the reference's coercion order and invalid-date
rules, and the string forms: deterministic ISO toString, en-US UTC locale
strings, host-local toDateString/toTimeString, toUTCString. Date.parse
follows JavaScriptCore: the ISO format first, then the legacy formats.
Dates take the string hint for default ToPrimitive and serialize through
toJSON.

Local time follows the host zone (TZ, else /etc/localtime) through the
tz-rs crate (pure Rust TZif reader, no dependencies); with this commit's
Date code the stripped embed example grows from 3178872 to 3262504 bytes.
…b and btoa

Uint8Array with JS byte semantics (wrapping writes, shared subarray views,
live iterators, the callback methods, base64 and hex), Node's Buffer over
it with its encodings, numeric reads and writes and Node error codes,
UTF-8 TextEncoder and streaming TextDecoder, and forgiving-base64 atob and
btoa. Also fixes Object.prototype.toLocaleString to call toString and adds
Array.prototype.toLocaleString. No new dependencies; the stripped embed
example grows from 3262504 to 3379344 bytes.

Uint8Array numeric arguments and byte values convert without consulting program methods, as the reference's coerceToNumber does.
…et methods

Iterator with Iterator.from and the lazy helpers (map, filter, take, drop,
flatMap) and consumers (reduce, toArray, forEach, some, every, find),
closing their source on early exit; Map getOrInsert(Computed), Map.groupBy,
Object.groupBy, the Set composition methods over Set-like operands, and
WeakMap/WeakSet. Map and Set methods now reject each other's receivers,
Iterator.prototype.next rejects foreign receivers, and exhausted built-in
iterators stay done, as in the reference.

RegExp: the flag accessors are listed on RegExp.prototype, and patterns
without u or v are matched as code units, so astral literals match.
No new dependencies.

test262: the reference's skipped.txt drops twelve Iterator and Set files the reference itself passes, which pass here too. Built-in iterators report their JS tags ([object Array Iterator], [object Set Iterator]), as the reference now does.
hasOwnProperty, isPrototypeOf and propertyIsEnumerable now throw the
reference's TypeError for undefined, null, primitives and tool references
instead of answering false, valueOf and toLocaleString throw on null and
undefined, and a tool reference's toString tag is [object Object], all as
in the reference.

reference-gaps.txt drops the four Promise files that passed here only through the accepted undefined receiver.
…6round and copyWithin

- encodeURI/encodeURIComponent/decodeURI/decodeURIComponent with the
  reference's URIError messages.
- Number formatting follows JavaScriptCore: toString(radix) ports its exact
  integer digits and shortest-fraction algorithm, toExponential and
  toPrecision round half up on the exact value, toFixed keeps the sign of
  a negative that rounds to zero, and number-to-string picks the even
  digit when two shortest forms tie. Range errors use the host messages.
- Math.sumPrecise sums exactly and rounds once; Math.f16round rounds
  straight from the double; Array.prototype.copyWithin.

Number-to-string and its tie rule live in value.rs, so integers from 2^53
up print their shortest digits instead of every digit.

copyWithin converts its arguments through ToPrimitive like the other Array methods. Number-to-string decides ties in 128-bit integer arithmetic. test262: the reference's skipped.txt drops three copyWithin files the reference passes.
…redClone

URL with its readable and writable fields and a linked searchParams,
URL.canParse/parse; URLSearchParams from strings, records, pairs and
other params, with live iterators; Headers with name and value
validation, combined values and set-cookie kept apart in the host's
order; crypto.randomUUID and getRandomValues from /dev/urandom; and
structuredClone over the data kinds, rejecting the rest with
DataCloneError. Writing a read-only accessor (RegExp flags, Uint8Array
length, size, TextDecoder fields, URL origin) throws the reference's
TypeError.

Adds the url crate (pure Rust WHATWG URL parser) with idna_adapter pinned
to 1.1, which maps IDNA with the unicode-rs crates instead of ICU4X; the
stripped embed example grows from 3512936 to 3879376 bytes.

Built-in iterators over URLSearchParams and Headers report the reference's tags.
…es them

RegExp, Date, the byte types, the Map/Set additions, the URI functions,
URL/URLSearchParams/Headers and crypto/structuredClone are set up when the
program names one of their globals, imports a module that maps to them, or
makes one of their values (a regex literal, a String regex method), so a
run that uses none of them starts as fast as before they existed.

Touches vm.rs (load installs the groups its globals need) and host.rs
(module imports install a group before falling back to host globals).
… spread, a call in flight or an iterator helper
…child enumeration that covers every VM reference
An incremental mark-sweep collector frees objects, closure cells and
strings at safe points, a bounded step at a time, paid for by
allocation. Marking keeps a snapshot of the heap as a cycle starts: the
first write to an object while marking traces it, a cell write shades
the value it overwrites, and what a cycle allocates is black. Long
objects are traced a chunk at a time. Runs that never reach the first
threshold do no collector work.

reach.rs reports strings and closure cells as well as objects, and the
collector marks from the same roots and children as snapshot
reachability. The realm's lazily made prototypes join the roots.

PSYCHOSCRIPT_GC_STRESS=1 collects at every safe point, in tiny steps.
…r use

An atom now lives exactly as long as its string. Marking an object marks
its keys' atoms, and names in natives, namespaces, globals and host
namespaces are traced too. Map and Set entries keep a string key as its
atom's string, as JSON.stringify's walk now does for the keys it holds,
so tracing the entry keeps the atom. While a cycle runs the interner
shades every atom it gives out. The sweep frees atoms whose strings were
not reached and the interner reuses their ids.

Atoms of the fresh runtime and of the compiled program stay permanent.
Snapshots save a freed atom as null and restore it as a free id.
OrderedMap never dropped its tombstones, so a Map or Set with constant churn
grew without bound. Each entry now has a logical position, numbered in
insertion order and never reused, and iterators, forEach and the Set methods
hold logical positions instead of slots, through OrderedMap::next_entry.

Once tombstones outnumber live entries (and 32), a delete compacts: tombstones
in front alone move the rest down, as index values count from a wrapping
shift; otherwise every survivor moves, keeping its position in a column used
only when positions have gaps. Snapshots save live entries with their
positions and the next one.
# Conflicts:
#	packages/codemode/src/language/generators.ts
#	packages/codemode/src/language/internal-prelude.ts
#	packages/codemode/src/language/prelude/bytecode.json
#	packages/codemode/src/language/runtime/errors.ts
#	packages/codemode/src/language/vm.ts
#	packages/codemode/test/test262/skipped.txt
Port d3f6559 (#64908) to psychoscript: object-literal get/set, accessor
properties and their descriptors (defineProperty, defineProperties,
getOwnPropertyDescriptor(s), getOwnPropertyNames, Object.create with
descriptors), program getters and setters called from property instructions,
built-ins reading getters ahead and running again (withReads), getters in
iteration, conversions, thenables, JSON, the result crossing and host-call
arguments, display labels, and a streaming response's live state as hidden
accessors. Accessor pairs are heap objects, traced and saved like any other.
Recordings regenerated from the reference; both test262 lists are empty.
… reference

New cases for accessor literals and the descriptor built-ins, host runs where
getters cross into host calls, park on a host wait or survive a suspend, and a
collector test for accessor halves replaced while cycles run.
# Conflicts:
#	packages/codemode/src/language/vm.ts
#	packages/codemode/test/test262/skipped.txt
…NT from #64930

Class declarations and expressions with constructors only new may call,
instance and static methods, accessors and fields, extends of classes and
Error types, super(...) and super.x reads, writes, calls, compound
assignments and deletes (also in object literal methods), home objects,
derived constructors' this before super() and their return rules,
Function.prototype's restricted caller and arguments, class names in
console.log, and the reference's texts for refused class syntax,
decorators and private names. An element read or written with an object
key now calls a program getter or setter, and a function under a computed
key is named by it. Uint8Array, its prototype and Buffer get
BYTES_PER_ELEMENT. #64919 changes only test hooks. Recordings regenerated
from the reference, with class cases from its tests.
…'s order

Object.getOwnPropertyNames of every built-in constructor and prototype now
lists the reference's names in its order: methods, constants and accessors
move to where the reference defines them, generator prototypes get their
read-only constructor, and Symbol has no prototype. A conformance case
records the names of all of them.
…etters in their arguments

A host function used as a callback, through call, apply or bind, or as a
promise handler reads its arguments' program getters first, as when
program code calls it; a getter that throws fails or rejects the call.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant