[WIP] Support ListExpression '@[]' in PowerShell - #5762

PR Summary

Another attempt to address #5643

This design is to add the ListExpression '@[]' to PowerShell, which is similar to ArrayExpression '@()' but always returns a List<object>.

Please review with #5761 as a comparison, where list is supported as a ListLiteralExpression, similar to ArrayLiteralExpression.

A blocking issue

@[] has a SubExpression like '@()' and '$()'. However, unlike the closing parenthesis character ')', the closing bracket character ']' doesn't always force to start a new token, and it can be included in a generic token, meaning that ']' can appear in a command name, argument, or function name. This makes it impossible for @[dir] to determine the ending of the list expression because dir] will be treated as a single generic token.

This PR adds the property InListSubExpression to Tokenizer, and makes ']' a force-to-start-new-token character when _tokenizer.InListSubExpression is set. This approach solves the most common UX problem but is by no way perfect, for example, comparing to @(funcHas[]inName) or @(dir has[]inpath), @[funcHas[]inName] and @[dir has[]inpath] won't work because the first ']' will force the command name to end.

I couldn't find a less hacky solution without breaking changes (make '[' and ']' always not included by GenericToken). So I started to think what if we support list in PowerShell like the ArrayLiteralExpression instead of @(). In that case, a list can only contain Expression elements and hence command name, arguments, and function names won't be a problem for the ending bracket. #5761 was also submitted to support list as a ListLiteralExpression.

Example

# an empty list
> $l = @[]
> $l.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     List`1                                   System.Object

# one element list, the element is a type
> $l = @[[string]]
> $l.Count
1
> $l[0]

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     String                                   System.Object

# multi-element list
> $l = @[$pid,2,$home,4]
> $l.Count
4
> $l.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     List`1                                   System.Object

# arbitrary statements in @[], like @()
> $l = @[dir]
> $l.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     List`1                                   System.Object

> $l.Count
30

PR Checklist

Note: Please mark anything not applicable to this PR NA.

• PR has a meaningful title
  • Use the present tense and imperative mood when describing your changes
• Summarized changes
• User facing Documentation needed
  • Issue filed - Issue link:
• Change is not breaking
• Make sure you've added a new test if existing tests do not effectively test the code changed
  • Add [feature] if the change is significant or affectes feature tests
• This PR is ready to merge and is not Work in Progress.
  • If the PR is work in progress, please add the prefix WIP: to the beginning of the title and remove the prefix when the PR is ready.