I recently ran a one-liner against a file that was much bigger than the ones I usually ran it against and, when that (obviously) took a lot longer to finish than I hoped, thought "I probably wouldn't mind so much, if this had a progress bar".

That turned out to be just annoying enough to implement, that I wondered, "surely there's a better way...?" I thought I'd start this discussion to see if we could come up with one...

Consider the following, pretty-standard, "objectify a log file and then do something with it" code. The log line format is unimportant for the discussion, but it's basically time,msgtype,sessionid,messagetext – assume the regex is suitable:

Select-String -Path .\20210124.log -Pattern '^(?<time>\d\d:\d\d:\d\d),(?<msgtype>\w+?)\s*,(?<session>\d+):\s*(?<message>.*?)\s*$' `
  | Select-Object -ExpandProperty Matches `
  | Select-Object -Property @{n="Time";    e={$_.Groups['time'].Value}},
                            @{n="Session"; e={$_.Groups['session'].Value}},
                            @{n="MsgType"; e={$_.Groups['msgtype'].Value}},                            
                            @{n="Message"; e={$_.Groups['message'].Value}} `
  | ForEach-Object {
        <# Arbitrary/Unimportant #>
    }

My first thought was "put a Write-Progress at the top of the ForEach-Object. That's OK but, now, how to determine the -PercentageComplete? Without it, you're still no clearer on how long the above is going to take. It's pretty basic maths – 100 * position / length – but you have no access to reasonable position or length values.

Select-String will yield the line number (position) but getting LineNumber means giving up on -ExpandProperty Matches and introducing a second foreach somewhere, so you can join up LineNumber and Match.

Select-String -Path .\20210124.log -Pattern '^...$' `
  | ForEach-Object { 
        $ln = $_.LineNumber
        $_.Matches `
            | Select-Object -Property @{n='LineNumber'; e={$ln}},
                                      @{n="Time";       e={$_.Groups['time'].Value}},
                                      @{n="Session";    e={$_.Groups['session'].Value}},
                                      @{n="MsgType";    e={$_.Groups['msgtype'].Value}},                            
                                      @{n="Message";    e={$_.Groups['message'].Value}} 
    }
  | ForEach-Object {
        Write-Progress -Activity "Reading" -Status "Reading" `
                       -PercentComplete (100.0 * $_.LineNumber / $OhIDontHaveALineCount)
        <# Arbitrary/Unimportant #>
    }

...and then, unless you scan the file twice, you don't have a total line count (length), so you can't get any further with that.

Maybe introduce Get-Item? That would get you access to a total byte length, without having to scan the file, and the Match.Index property is supposed to give the position of the match...

$file = Get-Item .\20210124.log
$file `
  | Select-String -Pattern '^...$' `
  | Select-Object -ExpandProperty Matches `
  | Select-Object -Property @{n="Position"; e={$_.Index}},
                            @{n="Time";     e={$_.Groups['time'].Value}},
                            @{n="Session";  e={$_.Groups['session'].Value}},
                            @{n="MsgType";  e={$_.Groups['msgtype'].Value}},                            
                            @{n="Message";  e={$_.Groups['message'].Value}} `
  | ForEach-Object {
        Write-Progress -Activity "Reading" -Status ($file.Name) `
                       -PercentComplete (100.0 * $_.Position / $file.Length)
        <# Arbitrary/Unimportant #>
    }

At first glance this seems OK...but no. Seems Select-String goes line-by-line (as it should), and so $_.Index is relative to the line, not the overall input. Since the Pattern is a ^...$ expression, the match is always the entire line, and $_.Index is always going to be 0. Besides, $_.Index is a character position, and $file.Length is a byte length, and one day encoding is going to come along and make those not play nicely together.

In then end, had to forego the idiomatic approach and settle on...

try {
    $sr = New-Object IO.StreamReader -ArgumentList .\20210124.log

    while (-not $sr.EndOfStream) {
        $line = $sr.ReadLine()

        $line `
          | Select-String -Pattern '^(?<time>\d\d:\d\d:\d\d),(?<msgtype>\w+?)\s*,(?<session>\d+):\s*(?<message>.*?)\s*$' `
          | Select-Object -ExpandProperty Matches `
          | Select-Object -Property @{n="Time";    e={$_.Groups['time'].Value}},
                                    @{n="Session"; e={$_.Groups['session'].Value}},
                                    @{n="MsgType"; e={$_.Groups['msgtype'].Value}},                            
                                    @{n="Message"; e={$_.Groups['message'].Value}} `
          | ForEach-Object {
                Write-Progress -Activity "Reading" -Status ($file.Name) `
                               -PercentComplete (100.0 * $sr.BaseStream.Position / $sr.BaseStream.Length)
                <# Arbitrary/Unimportant #>
            }
     }
}
finally {
    if ($sr) { $sr.Dispose() }
    Remove-Variable sr,line,file -ErrorAction SilentlyContinue
}

...and, while that isn't a drastic change from the original, it's still a shift from the more-idiomatic pipelined statement. A larger command or procedure might suffer:

• the .NET object I've had to instantiate myself, and that needs disposing
• the extra leaking $sr and $line variables I hope I'm not clobbering from somewhere (I could $private:-scope these, I suppose)
• the while loop instead of the idiomatic ForEach-Object

Reporting progress should be easy, and now I'm...sad...for the inelegance I've had to introduce. What started as a nice, pasteable snippet now has to be a function or script.

So, can anything be done to improve the progress-reporting experience in an idiomatic fashion?

I thought of a few ideas, but these come wholly from the perspective of a PS user, not a PS developer, and I have little input much effort something like this would take. Even as a user, I've pretty much talked myself out of all of them:

1. Have certain cmdlets emit a percentage property of their own?
   Would be relatively easy to drop into an existing script:

   Select-String -Path .\20210124.log -Pattern '^...$' `
     | ForEach-Object { Write-Progress -Activity 'Reading' -Status 'Just reading' -PercentComplete ($_.PercentComplete) } `
     | Select-Object -ExpandProperty Matches #...
   

   But that might work for Select-String, but what about other cmdlets? Get-Content would be an obvious candidate, but that doesn't return a structured object, and you don't want to break existing scripts by changing the output of something as fundamental as Get-Content.

2. Ratify a known variable, such as $PSProgress, which cmdlets can choose to update?
   Requires support from cmdlets, but seems more light-touch to my eye. Cmdlets that take a -Path argument could update $PSProgress if they know their length and position in advance; cmdlets that don't (or where input is piped) do not. Somewhere along the line, something might have updated $PSProgress, and we can get some visual feedback from that:

   Select-String -Path .\20210124.log -Pattern '^...$' `
     | ForEach-Object { Write-Progress -Activity 'Reading' -Status 'Just reading' -PercentComplete $PSProgress } `
     | Select-Object -ExpandProperty Matches #...
   

   That said, introducing new variables would affect anyone who had inadvertantly defined the same name already (variables named $PSSomething are not restricted, after all), and the global nature of the variable may lead one cmdlet to clobber the value for another – I could see Get-ChildItem setting one up for the files in a directory, then Get-Content immediately overwriting it, for example.

3. Ratify a -ShowProgress switch parameter, and introduce in certain cmdlets?
   "Solves" the Get-Content problem by not introducing an output property. "Solves" the clobbering problem by not relying on global state. Easy enough for a user to choose what shows progress and what doesn't:

   Select-String -Path .\20210124.log -Pattern '^...$' -ShowProgress `
     | Select-Object -ExpandProperty Matches #...
   

   But it's a lot of cmdlets to go through to add in support, and clutters up the implementation of those cmdlets with progress-reporting stuff. In general, this is the thing I end up adding to my own cmdlets, and it's always a pain to have to do.

4. Have Write-Progress take a pipeline input and emit the same object as output, and attempt to identify progress from that?
   I almost like this one, as it appears to localise the problem to Write-Progress, and again is easy enough for a user to invoke, by inserting into the pipeline where you want it. Shouldn't have backward-compatibility issues, as Write-Progress doesn't currently take input or yield output:

   Select-String -Path .\20210124.log -Pattern '^...$' `
     | Write-Progress -Activity 'Reading' -Status 'Just Reading' `
     | Select-Object -ExpandProperty Matches #...
   

   But I can't see how the implementation could be anything other than magic, and that probably isn't desirable. Stuff like if ($_ -is [IO.Stream]) { ... } elseif ($_ -is [ICollection]) { ... } ... would clutter the implementation of Write-Progress, and would probably need as much upstream support from individual cmdlets as the others, arguably needing something like (1) or (2) anyway, for the common case.

So I open it up to the community. I'm sure there are other ideas that I haven't considered, and would love to understand your take on them...