Repository navigation
Close the feature gap between the Where-Object cmdlet and the .Where() array method, introduce -First, -Last, ... switches #13834
Description
Summary of the new feature/enhancement
Update for clarification: The gist of this proposal is to bring valuable filter functionality that currently only exists in the in-memory .Where() array method to its pipeline counterpart, Where-Object.
(Always using .Where() isn't an option with large input sets, because you'd have to collect all input in memory first.)
The .Where() array method, which is the in-memory equivalent of the pipeline-based Where-Object cmdlet, offers a number of additional features, such as the ability to optionally stop matching after the first match is found, which can be an important optimization technique, and the ability to output all objects that come before/after a matching one:
First / -First example:
# OK - very fast, because - despite the large input collection - processing stops after the first match.
(1..1e6).Where({ $_ -eq 10 }, 'First') # -> 10
# WISHFUL THINKING: Add a -First switch (among several others - see below).
1..1e6 | Where-Object { $_ -eq 10 } -First
# Without it, ALL elements are processed - SLOW
1..1e6 | Where-Object { $_ -eq 10 }
# Current workaround is cumbersome.
1..1e6 | Where-Object { $_ -eq 10 } | Select-Object -First 1SkipUntil / -SkipUntil example:
(1, 2, 42, 43).Where({ $_ -eq 42 }, 'SkipUntil') # -> 42, 43
# WISHFUL THINKING: Add a -SkipUntil switch.
1, 2, 42, 43 | Where-Object { $_ -eq 42 } -SkipUntilBased on the current .Where() features, the following switches should be introduced with the same behavior, mirroring the WhereOperatorSelectionMode enumeration values:
-First, -Last, -SkipUntil, -Until, -Split
Note:
-
-Splitpartitions the input into two and returns two collections, so it would in effect require collecting all input in memory first before producing output, which would have to be clearly documented. -
The
.Where()method has an optional numeric parameter,numberToReturn, that modifies the operations, such as2withFirstreturning the first two matches; since we don't have a syntax for switches with optional (non-Boolean) arguments for commands (see Feature Request: Variable Parameter with switch-like capability #12104), we have the following options:-
Option A: Simply omit this aspect of the functionality and invariably default to
1, which is likely fine in the majority of cases; if a different number is needed,Select-Object -First/-Last $ncan be piped to. -
Option B: Implement
-Firstand-Last not as _switches, but as-Firstand-Last ` ; the potential downside is that a number argument is then mandatory. -
Option C: Given that
numberToReturnalso modifies all other functionality - though there's likely less of a need for that - implement a separate-Count <int>parameter, which in the absence of any of the switches would imply-First.
-
-
For these behaviors to be implemented efficiently, they have to stop the pipeline on demand, as
Select-Objectalready does. However, the latter does so in a problematic fashion - not giving other cmdlets a chance to run theirEndblocks - which should be addressed (independently) as well: see Select-Object -First X skips End{ } blocks of previous cmdlets #7930
On a meta note: You can skip the obsolete comments that follow and resume reading at this comment.
Activity
vexx32 commentedon Oct 22, 2020
Eh, I'd say the only reason
.Where()handles that is because we don't have a.Select()magic method.I don't see a real need to add this to
Where-Objectgiven thatSelect-Objectalready performs this function. 🤷mklement0 commentedon Oct 22, 2020
vexx32 commentedon Oct 22, 2020
Select-Object -First $xalready short-circuits the pipeline.Adding it to
Where-Objectas well would 1) duplicate the code path (unless you move that code somewhere common) and 2) not really gain a whole lot above that, I wouldn't think.mklement0 commentedon Oct 22, 2020
vexx32 commentedon Oct 22, 2020
No, I understand well enough.
I'm just saying I don't think that's meaningfully or usefully better than just doing
$stuff | Where-Object { $_ -eq 'bar' } | Select-Object -First 1mklement0 commentedon Oct 22, 2020
[-]Close the feature gap between the Where-Object cmdlet and Where array method, introduce -First, -Last switches[/-][+]Close the feature gap between the Where-Object cmdlet and Where array method, introduce -First, -Last, ... switches[/+]vexx32 commentedon Oct 22, 2020
In the case of Get-Content it's able to take additional shortcuts due to how the IO reading works.
For the case of whether there's a switch/parameter for Where-Object vs Select-Object I think there's even less reason to do it. If you really want it I guess I have nothing specifically against it, I just don't consider it worth the time to do. 🙂
mklement0 commentedon Oct 22, 2020
vexx32 commentedon Oct 22, 2020
Yeah, those are probably worth adding and more aligned with Where-Object's general role IMO.
mklement0 commentedon Oct 22, 2020
Rain Sallow (/u/ta11ow) (@vexx32), let me try to summarize, and, if we agree, I suggest we hide all previous comments:
The gist of this proposal is to bring valuable filter functionality that currently only exists in the in-memory
.Where()array method to its pipeline counterpart,Where-Object.(Always using
.Where()isn't an option with large input sets, because you'd have to collect all input in memory first.)The proposed new switches,
-First,-Last,-SkipUntil,-Until,-Splitwould directly correspond to theWhereOperatorSelectionModeoperation enumeration values (except forDefault) accepted by the.Where()method.You, Rain Sallow (/u/ta11ow) (@vexx32), disagree with this proposal in part:
-SkipUntil,-Until,-Splitis worth adding,-Firstand-Lastaren't needed, because you can pipe toSelect-Object -First 1andSelect-Object -Last 1instead and therefore do not strictly need to this functionality inWhere-Objectitself.I disagree with this for the following reasons:
Concision and convenience:
1..1e6 | Where-Object { $_ -eq 10 } -Firstis much shorter to type than1..1e6 | Where-Object { $_ -eq 10 } | Select-Object -First 1Get-Content -First $ninstead ofGet-Content | Select-Object -First $nandSelect-String -Rawinstead ofSelect-String | Select-Object -ExpandProperty LineSymmetry: Only adding some of the operations of the analogous
.Where()method is awkward.Performance: While the performance gain is negligible in the
-Firstcase, with-Lastit matters:Where-Objectcall, an internal-Lastimplementation could keep only the most recent match and replace it if another one comes around; by contrast, if the-Lastfunctionality is provided bySelect-Object,Where-Objectwould send all matches through the pipeline beforeSelect-Objectcan select the last.SeeminglyScience commentedon Oct 22, 2020
What about new commands?
Select-Object -First 1doesn't feel great interactively. That's why I havefirst,last,skip, andat(indexing) defined in my profile (Source).e.g.
Also would be nice to be able to stop the pipeline without killing it completely like
Select-Object -First 1does (well, an easier way then the nightmare of reflection and implementation detail abuse I linked above).KirkMunro commentedon Oct 22, 2020
If this moves forward, I'd love to see
-Firstand-Lastimplemented as<Switchable[int]>values, so that you don't need to provide a number if you are getting the very first one or the very last one. If you don't know what I mean, I proposed on another PR that there are scenarios where we might want a parameter to have a default value when used as a switch, or accept a specific value as input as well. This is one of those scenarios.Are these new parameters being proposed only for the
-ScriptBlockparameter sets? Or for all parameter sets. Maybe we could have a new poster boy for parameter set nightmares. Let's see...31 parameter sets times 5 new incompatible, optional switches per parameter set, so 135 parameter sets in total. 🤣Also FWIW, I would agree with all 5 new parameters to cover all options, especially given Michael Klement (@mklement0)'s argument for
-Last.mklement0 commentedon Oct 22, 2020
Patrick Meinecke (@SeeminglyScience), those are interesting ideas, but it's really a separate discussion.
Kirk Munro (@KirkMunro) :
😁 re parameter sets, though perhaps that will inspires us to improve matters there, such as implementing simpler mutual exclusion... - ideally, we wouldn't have to restrict this to the script-block parameter set solely for technical reasons.
Re
<Switchable[int]>- something like that was also proposed in #12104, which I mention in the OP (couldn't find the PR you're referring to).While I like the idea in general, in this particular case I'm leaning toward option C from the (updated) OP: A separate, optional
-Count <int>parameter, given that such a quantifier applies to all operations supported by the.Where()method - we would need that for full feature parity.11 remaining items
SeeminglyScience commentedon Oct 24, 2020
Come on man, is that really necessary? We can have a minor disagreement and still remain civil.
mklement0 commentedon Oct 24, 2020
Patrick Meinecke (@SeeminglyScience):
What is necessary is to address the issue of how this debate unfolded so as to inform future debates, and my comment was part of that.
My intent was not to be uncivil: perhaps I was dramatic in my (genuine) expression of my disappointment, but that was only the garnish to the arguments made, and I ask you to focus on that, just as am I trying to see past the "Ehs", "🤷", "Don't know what to tell ya"s, "Yeah, frankly"s, "good luck!"s, and "man"s - civility is in the eye of the beholder.
But, point taken: I will try without expressions of disappointment going forward.
So let us exhaust the exhaustion, and lay waste to the wastefulness:
Rain Sallow (/u/ta11ow) (@vexx32):
My bafflement was about something different: It was about first saying "let's do this" and then seamlessly transitioning to "good luck trying to do that!", and considering that to be the end of the discussion.
Again, having clarity on what should be done even if it currently cannot be done and to also communicate it that way is important - it is an investment in a potentially better future.
Conversely, the clarity on what should be done informs the investigation into whether it can currently be done, after all.
Concluding the discussion with "Good luck trying to do that!" thwarts both those goals.
Notice how Kirk Munro (@KirkMunro) was the first to bring up implementation concerns above, yet he commendably did not use that to dismiss the proposal.
Returning to the issue at hand and its implementation challenges:
The proliferation of parameter sets can be worked around by implementing the mutual exclusion inside the command - not great, but doesn't impact users much.
Re stopping the pipeline prematurely, which is necessary to implement the proposed behaviors efficiently: Agreed that it should be done in a way that gives the other cmdlets in the pipeline a chance to run their
Endblocks, as @SeeminglyScience has advocated, albeit in the context of an unrelated proposal that couldn't be used to implement the bulk of this proposal's functionality, but that too is a separate issue: Select-Object -First X skips End{ } blocks of previous cmdlets #7930SeeminglyScience commentedon Oct 24, 2020
You can express disappointment without being disrespectful. Specifically "than the I'm-dismissing-this-for-implementation-concerns-alone logic discussed above." is incredibly rude. I'm no stranger to folks being rude on the internet and typically I don't say anything, but if you actually want to have productive discussions that's a sure fire way to kill that possibility.
Folks are free to change their mind in a discussion. The 32 already existing parameter sets were probably not at the top of his mind. Also saying something "is probably worth adding" is not a promise to fight for the thing you want.
This is your issue. While it's certainly a nice bonus for folks to come up with an alternate solution to implement the thing you are asking for, it is not a requirement or even an expectation.
I want to point out that at no point did I actually dismiss anything. I proposed an alternate solution, and you dismissed it as irrelevant to the conversation completely. For the majority of the discussion following that, I was mainly asserting that it did not belong in a different issue.
More than that though, I don't know why you find the idea of dismissing an implementation due to concerns of said implementation to be so horrible. The benefit of a feature or change always has to out weigh the amount of effort and risk involved.
I know this applies to you as well, but please keep in mind none of us are being paid to be here. If someone chimes into a thread with an implementation concern, it's not fair for you to demand that they also debate you on the merits of the idea that they do not think is actionable.
mklement0 commentedon Oct 24, 2020
Like civility (and flippancy), disrespect is in the eye of the beholder. To me, "I'm-dismissing-this-for-implementation-concerns-alone" still seems like a condensed, but accurate summary of your comments.
I'm not taking responsibility for this being "incredibly rude", but I'm sorry that your perception of it as such caused you distress - it wasn't my intent.
Absolutely, but that wasn't my point at all.
That's not what I was asking for, though it would indeed be nice.
I've argued multiple times why your alternate solution isn't one (to recap again: you can't use it to implement the bulk of the functionality in this proposal - perhaps I should have led with that, but it seemed obvious to me), but there was no engagement on these points.
I don't find it horrible either - it may just be a fact of life, recognized as such after an earnest investigation - but, again, not my point.
What is fair is to expect and ask for - not demand - that implementation concerns be presented in a constructive manner, and a flippant about-face to "Good luck with that!" doesn't qualify.
Similarly, it is fair to ask that the merits of the proposal be evaluated separately, because even if something cannot get implemented now, it's still valuable to have clarity on what some day can be done, perhaps in a different context.
And it is perfectly fine for someone to only contribute to the former, constructively, without wanting to engage in the latter (which some may consider wasted effort) - but that's not what happened here.
A constructive discussion can be had that way, and those that agree that the proposal has merit can then discuss the actual implementation concerns to assess whether it can be done and, if so, with how much effort.
It's ultimately always a tradeoff, but it should be driven by clarity on the merits of the proposal. And only with an interest in the proposal is someone likely to expend energy investigating the implementation challenges.
Conversely, using a lack of interest / objection to the proposal to make a categorical claim of infeasibility is unhelpful.
Of course, the outcome may still be that implementation is currently not feasible, but that determination would be the result of a constructive collaborative process, not a foregone conclusion.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 23, 2023
This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.
[-]Close the feature gap between the Where-Object cmdlet and Where array method, introduce -First, -Last, ... switches[/-][+]Close the feature gap between the `Where-Object` cmdlet and the `.Where()` array method, introduce -First, -Last, ... switches[/+]