BasicJsonParser mishandles braces within string values - #52182

BasicJsonParser.tokenize() tracks the nesting of objects and lists while it splits the JSON on the commas that are not nested, so that a comma inside a string does not separate entries. The tracking of '{', '}', '[' and ']' was applied unconditionally, including for characters that appear within a string literal. An opening brace or bracket in a string value therefore incremented the nesting count, and the comma that follows the entry was no longer treated as a separator. The entries were merged into a single token and silently dropped.

For example:

• {"foo":"{","bar":"baz"} parsed as a single entry foo with the value {","bar":"baz, instead of foo -> { and bar -> baz
• ["{","baz"] parsed as a single element {","baz
• {"foo":"[","bar":"baz"} parsed as a single entry foo with the value [","bar":"baz

A closing brace or bracket within a string has the opposite effect: it drives the nesting count negative, which hides the comma that separates the entries of a sibling object.

• {"a":"}","b":{"c":"d","e":"f"}} mapped b to {c=d} and promoted e to an entry of the enclosing object, instead of a -> }, b -> {c=d, e=f}

Strings containing commas are already supported and covered by stringContainingComma, so braces and brackets within a string need to be handled in the same way. The tokenizer already tracks the in-string state in Tracked.VALUE; this commit uses it to only update the nesting count when the character is not part of a string.

GsonJsonParser and JacksonJsonParser both return the expected result for all of the inputs above.

The four new tests fail on main and pass with the change:

BasicJsonParserTests > stringContainingOpeningBrace() FAILED
BasicJsonParserTests > stringContainingOpeningBracket() FAILED
BasicJsonParserTests > listWithStringContainingOpeningBracket() FAILED
BasicJsonParserTests > stringContainingClosingBraceWithNestedMap() FAILED

To make sure the fix does not trade one set of inputs for another, the two parsers were exhaustively compared with GsonJsonParser over every combination of the characters {, }, [, ], , and a placed in the two string values of {"a":"X","b":"Y"} and ["X","Y"]. Of the 3698 inputs that Gson accepts, BasicJsonParser currently returns a wrong result for 1548 of them; with this change it returns the result Gson produces for all 3698, and no input that was previously correct changes its result.

./gradlew :core:spring-boot:checkFormatMain :core:spring-boot:checkFormatTest :core:spring-boot:checkstyleMain :core:spring-boot:checkstyleTest :core:spring-boot:test --tests "org.springframework.boot.json.*" passes.