Repository navigation
Conversation
This gives us access to the input/output schemas during macro expansion, which we'll need in order to know whether an output node is a leaf node or not. We could in principle get rid of all of the inner `rule!` applications now and just inline them directly, but to prevent churn I have opted not to do so at this time.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The missing yeast-schema Bazel dependency blocks builds, and changed Swift token text leaves existing test expectations and annotations out of sync.
5 open issues
What changed in this PR
This PR lets YEAST’s schema-aware rewrite rules give bare output leaf nodes the matched input’s content instead of defaulting to empty token values.
Changes:
- Track schema leaf kinds, resolve inherited content, and reject ambiguous nested bare leaves.
- Build inherited tokens from stored content or source text.
- Adopt schema-aware rules in Swift and add schema and runtime regression tests.
| File | Description |
|---|---|
unified/extractor/src/languages/swift/swift.rs |
Adopt rules! and simplify the super_expr translation. |
shared/yeast/tests/test.rs |
Test inheritance from parsed, synthetic, and empty content. |
shared/yeast/src/lib.rs |
Expose the matched root to token construction. |
shared/yeast/src/build.rs |
Construct tokens with inherited content. |
shared/yeast-schema/src/schema.rs |
Track and expose leaf-kind metadata. |
shared/yeast-schema/src/node_types_yaml.rs |
Populate and test leaf-kind metadata from YAML. |
shared/yeast-macros/src/parse.rs |
Load output schemas and resolve schema-aware rule items. |
shared/yeast-macros/src/lower.rs |
Emit inherited-token builder calls. |
shared/yeast-macros/src/lib.rs |
Document bare-leaf inheritance and nesting restrictions. |
shared/yeast-macros/src/ast.rs |
Resolve bare leaf templates and test validation. |
shared/yeast-macros/Cargo.toml |
Add the schema dependency. |
Cargo.lock |
Record the new dependency relationship. |
Review effort: Max
The `rules!` macro needs this information to decide whether a bare output node should inherit the value of the matched input node.
The `rules!` macro needs to inspect every rule before expansion so it can apply schema-aware transformations consistently. When we encounter an embedded `rule!` macro inside of `rules!`, we now unwrap it and lower it as if it were an inline rule.
A bare leaf used as a rule result has an obvious value: the value of the node matched by the rule. Using the output schema we can now distinguish leaves from fieldless structural nodes and mark top-level bare leaves as inherited content. For nested bare leaves, there's no obvious input node from which to inherit their value, so these still need to be handled explicitly.
The `rules!` macro now reads output schemas during expansion, so the proc-macro target needs `yeast-schema` under Bazel as well as Cargo.
Bare leaf rule results now inherit the matched source text. Existing unsupported Swift nodes previously had empty content, so this changed the control-flow test output even though their AST structure was unchanged. To fix this, we now give `unsupported_node` results explicit empty content, leaving `super_expr` as the intentional inherited leaf and preserving existing extraction behaviour.
tausbn
force-pushed
the
tausbn/yeast-emit-tokens-for-leaf-nodes-on-ast
branch
from
October 29, 2026 23:01
3c8be31 to
6f93249
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved content-inheritance bugs leave childless structural nodes with empty values despite available source text and give Swift labeled function-reference unsupported nodes unintended nonempty values.
2 open issues
5 resolved since last review
Review effort: Max
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Macro lowering drops explicit rule attributes, making gated rules unconditional, while a missed Swift rule changes unsupported-node values from empty strings to source text.
1 open issue
2 resolved since last review
Review effort: Max
tausbn
marked this pull request as ready for review
October 30, 2026 12:45
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For rules where we're matching some node, and mapping it to some leaf node in the output schema (e.g. mapping
(superExpr)to(super_expr)in Swift), we would previously produce a content-less node by default.With the changes in this PR, we now detect the case where the right hand side of a rule is a bare leaf node (no fields, no explicit content). In this case, we give it a new kind of content,
Inherited, which results in it being assigned the source range of whatever was matched as its content.Should be reviewed commit-by-commit. As an experiment, I tried making the commits very small this time around (though one of them still ended up with the bulk of the changes). Do let me know if this made it harder or easier to review.