In a larger sense, the spec doesn't talk about whitespace and tokenization.

Tokens should never span white space, and tokens should never be for white space.

That is, parsing (grammar time) should not take space into account, only a token stream that has already been normalized WRT space and not include space-based tokens. There has been some conflation of lexing and parsing pointed out in other issues.

E.g., this came up with the flawed idea of having - part of a number instead of as an operator, and then not being able to use - as an operator in front of a number.

I'm actually no longer sure this is the right fix. Compare with "old" C++ templates where you had to insert a space between nested templates ends to avoid confusing the tokenizer.

E.g. std::vector<std::unique_ptr<uint32_t> > That last space was needed to avoid confusing old compilers. The old compiler would otherwise see ">>" and think it was a shift operator.

oh wait:

$ clang++ x.cc -o x.o
x.cc:5:30: error: a space is required between consecutive right angle brackets (use '> >')
  std::vector<std::vector<int>> a;
                             ^~
                             > >
1 error generated.
$ clang++ --version
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin19.4.0
Thread model: posix

oh wait:

It was fixed in C++11. You shouldn't get that error with -std=c++11.

But, C++11 is using a more complex parsing scheme than simpler languages, and we shouldn't burden WGSL with that complexity. Not to mention there is a pre-processor in the middle of the tokenization for C++.

That's also why I originally raised the white-space issues in my review: that's all covered by the pre-processing rules, but WGSL doesn't have a pre-processor, leaving a vacuum to fill regarding white space.

100% agreed we don't want those parsing complexities.

Coming back to this to implement in Tint and it actually makes other bits of parsing more complicated. With this change, anywhere there is a GREATER_THAN (say vec4<vec2<f32>> as the example above we have to handle the fact that we get back a LOGICAL_SHIFT_RIGHT or ARITH_SHIFT_RIGHT instead of a GREATER_THAN and then, if we consume a portion of the token, shift the GREATER_THAN or LOGICAL_SHIFT_RIGHT back into the token stream.

This is doable, but is something other implementations will have to guard against as well.

I'm sure (and from some discussion above) you don't want this complexity in a simple portable language. I think you'll find more corner cases, and different implementations might find different ones.

There are 3 options I see. You either accept this complexity, allow '> > >' to mean '>>>' or change how the parser/lexer works so the parser asks for specific tokens from the lexer instead of what's next.

Having the grammar as written in this CL is probably the right thing as '> > >' meaning '>>>' would be weird. But, implementers need to make sure they deal with the side effects.