PowerShell should support creating an List similar to how it supports arrays

Powershell supports creating arrays with $array = 'a', 1, '3' . Then you can add an element to the array with $array += 4, but this creates a new array which is not performant.

Powershell should have a syntax which allows creating lists.
Assuming the operator is @[...], you could create a list with $list = @['a', 1, '3'] and then you could add an element to the existing list with $list += 4 without PowerShell having to create a new list.
Note: this new operator might function more like @(...)

This design assumes that changing , would be a breaking change. I'm open to discussing changing , as well.

I filed this based on an offline discussion about this comment on a PR: #5625 (comment)

I like the idea but to truly impact performance you'd need to be operating on large lists. For a convenient way to create large lists, I would expect something like this to work @[Get-ChildItem C:\Windows -r -file *.dll -ea 0]. While the list literal form is nice, I don't see folks creating lists large enough to gain much of a perf benefit over using an array. Well, unless the list literal is created inside a busy (large n) loop.

Two points:

@() says - make sure the thing inside is an array. It's not necessary if you use , because the comma operator always creates an array.
I've often wondered if the comma operator could create a list instead of an array. I have a feeling most scripts would never notice a difference because of how freely things are converted to an object array.

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr) I updated my description based on Jason Shirk (@lzybkr) 's comments

I like the idea of having a list literal in powershell. I think it could have a syntax like @[1, 2, 3] to directly create a list with elements 1, 2 and 3 without first create an array literal from 1,2,3 and then make it a list using @[].

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr) is it something like this?
1,2,3
(1,2,3).length
( , (1,2,3) ).length
( @(1,2,3) ).length

Sure, alias properties would be needed to make lists work just like arrays.

Keith Hill (@rkeithhill)

I've often wondered if the comma operator could create a list instead of an array.

If that's not considered too much of a breaking change, it would certainly be the best solution.

Otherwise:

Keith Hill (@rkeithhill):

I get what you're saying about large lists, but that's where += comes in as a convenient syntax for appending to the list (calling ::Add() or ::AddRange() on the [System.Collections.ArrayList] or [System.Collections.Generic.List[object]] instance behind the scenes - unlike today's behavior of +=, which either silently recreates the variable content as an array or, if the variable was type-constrained, as a new instance).

In other words: something like the following would make sense:

$al = @[] # simpler than: [System.Collections.ArrayList]::new()

for ($i = 0; $i -lt 1000; ++$i) {
  $al += $i # simpler than: $null = $al.Add($i)
}

I submitted two PRs (WIP) with different designs for the List support in PowerShell:

[WIP] Support ListExpression '@[]' in PowerShell #5762 -- Support @[], similar to @()
[WIP] Support ListLiteralExpression '[]' in PowerShell #5761 -- Support ListLiteralExpression '[]', similar to ArrayLiteralExpression

@[] is my first design. However, I ran into a blocking issue regarding the closing bracket character ']'. Quoted from #5762:

@[] has a SubExpression like '@()' and '$()'. However, unlike the closing parenthesis character ')', the closing bracket character ']' doesn't always force to start a new token, and it can be included in a generic token, meaning that ']' can appear in a command name, argument, or function name. This makes it impossible for @[dir] to determine the ending of the list expression because dir] will be treated as a single generic token.

This PR adds the property InListSubExpression to Tokenizer, and makes ']' a force-to-start-new-token character when _tokenizer.InListSubExpression is set. This approach solves the most common UX problem but is by no way perfect, for example, comparing to @(funcHas[]inName) or @(dir has[]inpath), @[funcHas[]inName] and @[dir has[]inpath] won't work because the first ']' will force the command name to end.

Without breaking change, I think the best we can do is probably to make ']' a force-to-start-new-token character when parsing a command invocation pipeline in @[] but not when parsing any nested expression or statement within the @[].

At the same time, I started to think an alternative -- add ListLiteralExpression like the ArrayLiteralExpression. In that case, a list can only contain Expression elements and hence command name, arguments, and function names won't be a problem for the ending bracket. PR #5761 is for that design, where we use '[]' (same token pair as TypeConstraint and Attribute).

I hope the those 2 PRs can draw more discussion on the design.

Dongbo Wang (@daxian-dbw)

Dongbo Wang (@daxian-dbw) since this code is beyond my understanding, does it attempt to create a strongly typed list, or is always List<Object>?

Mark Kraus (@markekraus)

Mark Kraus (@markekraus) It attempts to always create List<object>, like @() ways create an object[].

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr) proposed to use new token pairs instead of @[] to represent a ListExpression in #5762 (comment):

You could consider 2 character tokens.
For example, F# uses this syntax for an array literal:
[| 1; 2 |]
There are other possibilities that probably aren't breaking changes, e.g. [< 1, 2 >].
The key here is to use a second character that can't be in a command name.

It would be great to have @[] to represent ListExpression, but I'm fine with new token pairs. ~~I will prototype with [<>]~~. [<>] won't work because '>]' is allowed in a generic token. [| .. |] may work. If new token pairs are acceptable, I definitely prefer ListExpression over ListLiteral.

Dongbo Wang (@daxian-dbw)

Dongbo Wang (@daxian-dbw): I'm really glad to see you take this on, but before we go any further with the syntax debate:

Is the consensus that we cannot just simply switch ,, the array construction operator, to an array-list/generic-list implementation behind the scenes, as Jason Shirk (@lzybkr) hinted at - for reasons of backward compatibility?

The answer may well be that yes, it's too risky to make that change (I personally cannot tell), but if it happens to be no, after all, there's no need for a syntax debate.

Michael Klement (@mklement0)

Michael Klement (@mklement0) IMHO, there would be 3 problems if we simply change the comma operator ',' to return a list:

The AST type name ArrayLiteralAst would be inconsistent, but changing it would be a huge breaking change. There would be other breaking changes like the returned value of StaticType property, but the AST type name would be the most problematic one I guess.
With the comma operator, we wouldn't be able to create an empty list.
The comma operator only takes Expression elements, not arbitrary statements like @() does, for example, comparing to @(dir), you would have to use ,(dir). Besides, the comma operator doesn't unwrap the Expression value because it's literal (ArrayLiteralAst). So ,(dir) would return a one-element list that contains an object array.

I prefer a ListExpressionAst '@[]' over a ListLiteralAst '[]' because of the 3rd one above.

Dongbo Wang (@daxian-dbw)

Dongbo Wang (@daxian-dbw) Thanks for great prototypes!
I'd prefer @[] if it would possible to implement. I very wonder to see something like[| ... |] - if we haven't another way I'd rather see simple List( ... ) or [List]1,2,3.

If we have problem with last ] in@[] could we use @[ 1, 2, 3 ]@ like multiline string literals?
@@[] don't resolve the problem.
We could reuse parentheses with other prefix - if @() array, $() singletion then %() or &() or *() - list.

I personally like *().

Reviving this. Rather than introduce a new language syntax, I think it would be simpler to just introduce the proposed [list] type accelerator which produces a [system.collections.generic.list[object]].

... If only that the implication of this syntactic sugar proposal will likely also support Constrained Language Mode.
See also: Mutual lists in Constrained Language Mode

The type accelerator doesn't change behavior of related operators.

I meant to comment on this thread but apparently never did. Specifically, regarding making += work that would be the first instance of += actually mutating the object on the LHS rather than returning something new.

With regards to List<> specifically, I think we'd be adding another common pitfall if it sees widespread use. The mutability of List<> is partially why you don't see it a lot in public API surfaces. It's so easy to slip up in this regard there's even an instance of mutating the caller's list in one of our public APIs (#12928).

That said, I did prototype (two years ago apparently) a List<>-y implementation with comparable Add performance, but with some extra guards against mutation. Definitely not ready to be added as is, but it's an approach with considering.

As a side note on the topic of lists and their place in PowerShell, they aren't actually very frequently better than simply using $myCollection = foreach ($a in $b) { }. The engine handling it for you is almost always better, so the amount of use cases for List<> specifically aren't actually as high as they appear. Granted not always an option, I'm not saying scenarios like adding across named blocks don't exist, just that they are not as frequent.

Patrick Meinecke (@SeeminglyScience)

Good point that using list types is often not needed, Patrick Meinecke (@SeeminglyScience).
iRon7 created a "canonical" answer on Stack Overflow that advises against += and preaches the gospel of statements and pipelines as expressions (where the engine automatically collects multiple outputs in an [object[]] array for you).

As for your "List<>-y" implementation. At least at first glance it sounds similar to what PetSerAl - who agrees that += shouldn't mutate the LHS (as do I now) attempted in the context of Assigning the result of an addition (+ operator) with an IList LHS back to that LHS should preserve the list (collection) type (#5805), specifically here.

Implementing our own list type that plays nicely with += without sacrificing (lots of) performance would provide two additional benefits:

Potentially providing syntactic sugar for construction, as previously discussed (@[...] or ....) - though finding a consensus on the syntax may be challenging.
Being able to avoid the pitfall mentioned by Staffan Gustafsson (@powercode), namely the List<T>'s native's .ForEach() method shadowing its intrinsic (engine-provided) counterpart.

The least-effort alternative, perhaps acceptable in light of the need for lists not being as pressing as it may seem, would be:

Simply provide type accelerators for existing list types, say [arraylist] and [list[object]] - though I'm not sure if the latter, with its generic parameter, fits into the current type-accelerator mechanism.
Consider their use an advanced use case and expect users to know how those types work and their pitfalls:
- the need to use .Add(), and additionally for [arraylist], to suppress the usually unwanted return value.
- that for [list[T]] .ForEach() isn't PowerShell's .ForEach() method
By contrast, [list], while easier to type, wouldn't readily reveal its relationship with the List<Object> type it represents, and potentially increase the risk of falling into the .ForEach() pitfall.
An unpleasant pitfall that applies to any type-literal / cast-based solution is that an array on the RHS must be (...)-enclosed, though using a type constraint avoids the problem:

using namespace System.Collections.Generic

$list = [List[int]] 1..10 # !! WRONG

$list = [List[int]] (1..10) # OK
[List[int]] $list = 1..10 # OK

PetSerAl

As for your "List<>-y" implementation. At least at first glance it sounds similar to what PetSerAl - who agrees that += shouldn't mutate the LHS (as do I now) attempted in the context of Assigning the result of an addition (+ operator) with an IList LHS back to that LHS should preserve the list (collection) type (#5805), specifically here.

Hah! They beat me to it by two years, that's amazing. Thanks for the link ❤️

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Navigation Menu

Repository navigation

PowerShell should support creating an List similar to how it supports arrays #5643

62 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PowerShell should support creating an List similar to how it supports arrays #5643

Description

Activity

rkeithhill commented on Dec 6, 2017

lzybkr commented on Dec 6, 2017

TravisEz13 commented on Dec 6, 2017

daxian-dbw commented on Dec 7, 2017

cchiu1979 commented on Dec 7, 2017

lzybkr commented on Dec 7, 2017

mklement0 commented on Dec 15, 2017

daxian-dbw commented on Dec 29, 2017

markekraus commented on Dec 29, 2017

daxian-dbw commented on Dec 29, 2017

daxian-dbw commented on Dec 30, 2017

mklement0 commented on Dec 30, 2017

daxian-dbw commented on Dec 31, 2017

iSazonov commented on Jan 1, 2018

62 remaining items

SteveL-MSFT commented on Oct 15, 2022

iRon7 commented on Oct 19, 2022

iSazonov commented on Oct 19, 2022

SeeminglyScience commented on Oct 19, 2022

mklement0 commented on Oct 21, 2022

SeeminglyScience commented on Oct 21, 2022

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 23, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions