Skip to content

WGSL grammar improvements #5461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
benbaarber wants to merge 4 commits into
base: main
Choose a base branch
from
+111 −40
Open

WGSL grammar improvements #5461

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3ef3054
wgsl grammar improvements
benbaarber Nov 27, 2025
06ae81b
revert leading underscore deletion for template_args
benbaarber Nov 28, 2025
07b5383
Merge remote-tracking branch 'origin/main' into wgsl-grammar-improvem…
benbaarber Nov 28, 2025
e385d5e
grammar.js postprocessing script for tree-sitter-wgsl
benbaarber Nov 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions wgsl/index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1535,8 +1535,8 @@ Note: For example, the phrase `var<storage,read_write>` modifies the general `va
</div>

The `'<'` (U+003C) and `'>'` (U+003E) code points that delimit a template list are also used when spelling:
* A comparison operator in a [=syntax/relational_expression=].
* A shift operator in a [=syntax/shift_expression=].
* A [=syntax/comparison_operator=] in a [=syntax/relational_expression=].
* A [=syntax/shift_operator=] in a [=syntax/shift_expression=].
* A [=syntax/compound_assignment_operator=] for performing a shift operation followed by an assignment.

The syntactic ambiguity is resolved in favour of template lists:
Expand Down Expand Up @@ -1603,7 +1603,7 @@ Let |TemplateList| be a record type containing:
* Advance |CurrentPosition| past this code point, and start the next iteration of the loop.
* If `'='` (U+003D) appears at |CurrentPosition|, then:
* Note: From assumption 1, no template parameter starts with `'='` (U+003C), so the previous code point cannot be the start of a template list.
Assume the current and previous code point form a <a for=syntax_sym lt=less_than_equal>`'<='`</a> comparison operator.
Assume the current and previous code point form a <a for=syntax_sym lt=less_than_equal>`'<='`</a> [=syntax/comparison_operator=].
Skip over the `'='` (U+003D) code point so a later step does not mistake it for an assignment.
* Pop the top entry from the |Pending| stack.
* Advance |CurrentPosition| past this code point, and start the next iteration of the loop.
Expand All @@ -1618,7 +1618,7 @@ Let |TemplateList| be a record type containing:
* Otherwise, this code point does not end a template list:
* Advance |CurrentPosition| past this code point.
* If `'='` (U+003D) appears at |CurrentPosition| then:
* Note: Assume the current and previous code points form a <a for=syntax_sym lt=greater_than_equal>`'>='`</a> comparison operator.
* Note: Assume the current and previous code points form a <a for=syntax_sym lt=greater_than_equal>`'>='`</a> [=syntax/comparison_operator=].
Skip over the `'='` (U+003D) code point so a later step does not mistake it for an assignment.
* Advance |CurrentPosition| past this code point.
* Start the next iteration of the loop.
Expand All @@ -1634,14 +1634,14 @@ Let |TemplateList| be a record type containing:
* If `'!'` (U+0021) appears at |CurrentPosition| then:
* Advance |CurrentPosition| past this code point.
* If `'='` (U+003D) appears at |CurrentPosition| then:
* Note: Assume the current and previous code points form a <a for=syntax_sym lt=not_equal>`'!='`</a> comparison operator.
* Note: Assume the current and previous code points form a <a for=syntax_sym lt=not_equal>`'!='`</a> [=syntax/comparison_operator=].
Skip over the `'='` (U+003D) code point so a later step does not mistake it for an assignment.
* Advance |CurrentPosition| past this code point.
* Start the next iteration of the loop.
* If `'='` (U+003D) appears at |CurrentPosition| then:
* Advance |CurrentPosition| past this code point.
* If `'='` (U+003D) appears at |CurrentPosition| then:
* Note: Assume the current and previous code points form a <a for=syntax_sym lt=equal_equal>`'=='`</a> comparison operator.
* Note: Assume the current and previous code points form a <a for=syntax_sym lt=equal_equal>`'=='`</a> [=syntax/comparison_operator=].
Skip over the `'='` (U+003D) code point so a later step does not mistake it for an assignment.
* Advance |CurrentPosition| past this code point, and start the next iteration of the loop.
* Note: Assume this code point is part of an assignment, which cannot appear as part of an expression, and therefore cannot appear in a template list.
Expand Down Expand Up @@ -6815,9 +6815,15 @@ path: syntax/additive_operator.syntax.bs.include
path: syntax/shift_expression.syntax.bs.include
</pre>
<pre class=include>
path: syntax/shift_operator.syntax.bs.include
</pre>
<pre class=include>
path: syntax/relational_expression.syntax.bs.include
</pre>
<pre class=include>
path: syntax/comparison_operator.syntax.bs.include
</pre>
<pre class=include>
path: syntax/short_circuit_and_expression.syntax.bs.include
</pre>
<pre class=include>
Expand Down
26 changes: 17 additions & 9 deletions wgsl/syntax.bnf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,18 +339,17 @@ additive_operator :

shift_expression :
additive_expression
| unary_expression _shift_left unary_expression
| unary_expression _shift_right unary_expression
| unary_expression shift_operator unary_expression
;

shift_operator :
_shift_left
| _shift_right
;

relational_expression :
shift_expression
| shift_expression _less_than shift_expression
| shift_expression _greater_than shift_expression
| shift_expression _less_than_equal shift_expression
| shift_expression _greater_than_equal shift_expression
| shift_expression '==' shift_expression
| shift_expression '!=' shift_expression
| shift_expression comparison_operator shift_expression
;

short_circuit_and_expression :
Expand Down Expand Up @@ -413,6 +412,15 @@ compound_assignment_operator :
| _shift_left_assign
;

comparison_operator :
'=='
| '!='
| _less_than
| _less_than_equal
| _greater_than
| _greater_than_equal
;

increment_statement :
lhs_expression '++'
;
Expand Down Expand Up @@ -559,7 +567,7 @@ function_decl :
;

function_header :
'fn' ident '(' param_list ? ')' ( '->' attribute * template_elaborated_ident ) ?
'fn' ident '(' param_list ? ')' ( '->' attribute * type_specifier ) ?
;

param_list :
Expand Down
15 changes: 15 additions & 0 deletions wgsl/syntax/comparison_operator.syntax.bs.include
View file
Open in desktop
Copy link
Copy Markdown
Member

@mehmetoguzderin mehmetoguzderin Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of creating a new rule for comparison operator, is it possible to keep them as before but just remove underline? This would reduce the diff of this PR and make it easier to review. It seems that removing preceding underlines from the rules uniformly could enable the use case, though it is important to make sure that scanner works well with the adjustments.

Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
<div class='syntax' noexport='true'>
<dfn for=syntax>comparison_operator</dfn> :

<span class="choice"></span> <a for=syntax_sym lt=equal_equal>`'=='`</a>

<span class="choice">|</span> <a for=syntax_sym lt=not_equal>`'!='`</a>

<span class="choice">|</span> [=syntax_sym/_less_than=]

<span class="choice">|</span> [=syntax_sym/_less_than_equal=]

<span class="choice">|</span> [=syntax_sym/_greater_than=]

<span class="choice">|</span> [=syntax_sym/_greater_than_equal=]
</div>
2 changes: 1 addition & 1 deletion wgsl/syntax/function_header.syntax.bs.include
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<div class='syntax' noexport='true'>
<dfn for=syntax>function_header</dfn> :

<span class="choice"></span> `'fn'` [=syntax/ident=] <a for=syntax_sym lt=paren_left>`'('`</a> [=syntax/param_list=] ? <a for=syntax_sym lt=paren_right>`')'`</a> ( <a for=syntax_sym lt=arrow>`'->'`</a> [=syntax/attribute=] * [=syntax/template_elaborated_ident=] ) ?
<span class="choice"></span> `'fn'` [=syntax/ident=] <a for=syntax_sym lt=paren_left>`'('`</a> [=syntax/param_list=] ? <a for=syntax_sym lt=paren_right>`')'`</a> ( <a for=syntax_sym lt=arrow>`'->'`</a> [=syntax/attribute=] * [=syntax/type_specifier=] ) ?
</div>
12 changes: 1 addition & 11 deletions wgsl/syntax/relational_expression.syntax.bs.include
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,5 @@

<span class="choice"></span> [=syntax/shift_expression=]

<span class="choice">|</span> [=syntax/shift_expression=] [=syntax_sym/_less_than=] [=syntax/shift_expression=]

<span class="choice">|</span> [=syntax/shift_expression=] [=syntax_sym/_greater_than=] [=syntax/shift_expression=]

<span class="choice">|</span> [=syntax/shift_expression=] [=syntax_sym/_less_than_equal=] [=syntax/shift_expression=]

<span class="choice">|</span> [=syntax/shift_expression=] [=syntax_sym/_greater_than_equal=] [=syntax/shift_expression=]

<span class="choice">|</span> [=syntax/shift_expression=] <a for=syntax_sym lt=equal_equal>`'=='`</a> [=syntax/shift_expression=]

<span class="choice">|</span> [=syntax/shift_expression=] <a for=syntax_sym lt=not_equal>`'!='`</a> [=syntax/shift_expression=]
<span class="choice">|</span> [=syntax/shift_expression=] [=syntax/comparison_operator=] [=syntax/shift_expression=]
</div>
4 changes: 1 addition & 3 deletions wgsl/syntax/shift_expression.syntax.bs.include
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,5 @@

<span class="choice"></span> [=syntax/additive_expression=]

<span class="choice">|</span> [=syntax/unary_expression=] [=syntax_sym/_shift_left=] [=syntax/unary_expression=]

<span class="choice">|</span> [=syntax/unary_expression=] [=syntax_sym/_shift_right=] [=syntax/unary_expression=]
<span class="choice">|</span> [=syntax/unary_expression=] [=syntax/shift_operator=] [=syntax/unary_expression=]
</div>
7 changes: 7 additions & 0 deletions wgsl/syntax/shift_operator.syntax.bs.include
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
<div class='syntax' noexport='true'>
<dfn for=syntax>shift_operator</dfn> :

<span class="choice"></span> [=syntax_sym/_shift_left=]

<span class="choice">|</span> [=syntax_sym/_shift_right=]
</div>
20 changes: 10 additions & 10 deletions wgsl/tools/extract-grammar.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

# TODO: Source from spec
derivative_patterns = {
"_comment": "/\\/\\/.*/",
"line_comment": "/\\/\\/.*/",
"_blankspace": "/[\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d\\u0085\\u200e\\u200f\\u2028\\u2029]/u"
}

Expand Down Expand Up @@ -639,15 +639,15 @@ class ScanResult(dict):
A dictionary mapping the name of an example to the
WGSL source text for the example.
The name is taken from the "heading" attriute of the <div> element.
self['_reserved']
self['reserved']
A list of reserved words.
"""
def __init__(self):
self[scanner_rule.name()] = dict()
self[scanner_example.name()] = dict()
self[scanner_token.name()] = dict()
self['raw'] = []
self['_reserved'] = []
self['reserved'] = []


def read_spec(options):
Expand All @@ -658,7 +658,7 @@ def read_spec(options):

# Read reserved words
with open('wgsl.reserved.plain', "r") as file:
result['_reserved'] = [s.strip() for s in file.readlines()]
result['reserved'] = [s.strip() for s in file.readlines()]

# Get the input bikeshed text.
scanner_lines = read_lines_from_file(
Expand Down Expand Up @@ -1002,7 +1002,7 @@ def flow_extract(options, scan_result):
],

extras: $ => [
$._comment,
$.line_comment,
$._block_comment,
$._blankspace,
],
Expand Down Expand Up @@ -1067,7 +1067,7 @@ def not_token_only(value):


for key, value in rules.items():
if key.startswith("_") and key != "_comment" and key != "_blankspace" and key not in rule_skip:
if key.startswith("_") and key != "line_comment" and key != "_blankspace" and key not in rule_skip:
grammar_source += grammar_from_rule(key, value) + ",\n"
rule_skip.add(key)

Expand All @@ -1083,9 +1083,9 @@ def not_token_only(value):


grammar_source += grammar_from_rule(
"_comment", {'type': 'pattern',
'value': derivative_patterns["_comment"]}) + ",\n"
rule_skip.add("_comment")
"line_comment", {'type': 'pattern',
'value': derivative_patterns["line_comment"]}) + ",\n"
rule_skip.add("line_comment")


# Extract space
Expand All @@ -1099,7 +1099,7 @@ def not_token_only(value):

# Reserved words

grammar_source += "\n _reserved: $ => choice('" + "', '".join(scan_result['_reserved']) + "'),\n"
grammar_source += "\n reserved: $ => choice('" + "', '".join(scan_result['reserved']) + "'),\n"

grammar_source += r"""
}
Expand Down
47 changes: 47 additions & 0 deletions wgsl/tools/process-ts-grammar.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""
Postprocessing grammar.js for tree-sitter-wgsl repository

Exposes block_comment, template_args_start, and template_args_end as named nodes
"""

import argparse
import sys


def main():
parser = argparse.ArgumentParser(
description="Postprocessing for grammar.js for tree-sitter-wgsl"
)
parser.add_argument("input", help="Input file path")
parser.add_argument("output", help="Output file path")

args = parser.parse_args()

try:
with open(args.input, "r", encoding="utf-8") as f:
content = f.read()
except FileNotFoundError:
print(f"Error: Input file '{args.input}' not found", file=sys.stderr)
return 1
except Exception as e:
print(f"Error reading input file: {e}", file=sys.stderr)
return 1

content = content.replace("_block_comment", "block_comment")
content = content.replace("_template_args_start", "template_args_start")
content = content.replace("_template_args_end", "template_args_end")

try:
with open(args.output, "w", encoding="utf-8") as f:
f.write(content)
print(f"Wrote processed file to '{args.output}'")
except Exception as e:
print(f"Error writing output file: {e}", file=sys.stderr)
return 1

return 0


if __name__ == "__main__":
sys.exit(main())