Skip to content

Add GitHub Actions annotations for Pester test failures #26789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Travis Plunk (TravisEz13) merged 9 commits into from
Feb 12, 2026
Merged

Add GitHub Actions annotations for Pester test failures #26789

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
739b0cc
Initial plan
Copilot Feb 12, 2026
d71c239
Add GitHub workflow annotations for Pester test failures
Copilot Feb 12, 2026
6a2d1af
Improve stack trace parsing for Pester 5 format
Copilot Feb 12, 2026
6605c89
Fix PSScriptAnalyzer warning: rename parameter from $StackTrace to $S…
Copilot Feb 12, 2026
9b85763
Convert file paths to relative paths for GitHub Actions annotations
Copilot Feb 12, 2026
790ef44
Add demonstration test for GitHub Actions annotations feature
Copilot Feb 12, 2026
3138c65
Replace demonstration test to use actual CI code path
Copilot Feb 12, 2026
b490e52
Move annotation generation to process-pester-results.ps1
Copilot Feb 12, 2026
0366a8b
Delete test/powershell/Modules/DemoGitHubAnnotations.Tests.ps1
TravisEz13 Feb 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .github/actions/test/process-pester-results/process-pester-results.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ $testIgnoredCount = 0
$testSkippedCount = 0
$testInvalidCount = 0

# Process test results and generate annotations for failures
Comment thread
TravisEz13 marked this conversation as resolved.
Get-ChildItem -Path "${TestResultsFolder}/*.xml" -Recurse | ForEach-Object {
$results = [xml] (get-content $_.FullName)

Expand All @@ -35,6 +36,61 @@ Get-ChildItem -Path "${TestResultsFolder}/*.xml" -Recurse | ForEach-Object {
$testIgnoredCount += [int]$results.'test-results'.ignored
$testSkippedCount += [int]$results.'test-results'.skipped
$testInvalidCount += [int]$results.'test-results'.invalid

# Generate GitHub Actions annotations for test failures
# Select failed test cases
if ("System.Xml.XmlDocumentXPathExtensions" -as [Type]) {
$failures = [System.Xml.XmlDocumentXPathExtensions]::SelectNodes($results.'test-results', './/test-case[@result = "Failure"]')
}
else {
$failures = $results.SelectNodes('.//test-case[@result = "Failure"]')
}

foreach ($testfail in $failures) {
$description = $testfail.description
$testName = $testfail.name
$message = $testfail.failure.message
$stack_trace = $testfail.failure.'stack-trace'

# Parse stack trace to get file and line info
$fileInfo = Get-PesterFailureFileInfo -StackTraceString $stack_trace

if ($fileInfo.File) {
# Convert absolute path to relative path for GitHub Actions
$filePath = $fileInfo.File

# GitHub Actions expects paths relative to the workspace root
if ($env:GITHUB_WORKSPACE) {
$workspacePath = $env:GITHUB_WORKSPACE
if ($filePath.StartsWith($workspacePath)) {
$filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
# Normalize to forward slashes for consistency
$filePath = $filePath -replace '\\', '/'
}
}

Comment on lines +65 to +71

Copilot AI Feb 12, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workspace-relative path conversion is brittle across platforms: StartsWith($workspacePath) is case-sensitive and doesn't normalize path separators, so Windows paths (or mixed \ vs /) may not match and will stay absolute, which GitHub annotations typically won't link correctly. Consider normalizing both paths (full path + separator normalization, OrdinalIgnoreCase on Windows), and always normalize to forward slashes even for already-relative paths.

Suggested change
if ($filePath.StartsWith($workspacePath)) {
$filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
# Normalize to forward slashes for consistency
$filePath = $filePath -replace '\\', '/'
}
}
try {
$workspaceFullPath = [System.IO.Path]::GetFullPath($workspacePath)
}
catch {
$workspaceFullPath = $workspacePath
}
try {
$fileFullPath = [System.IO.Path]::GetFullPath($filePath)
}
catch {
$fileFullPath = $filePath
}
# Normalize separators for comparison
$normalizedWorkspace = $workspaceFullPath.TrimEnd([System.IO.Path]::DirectorySeparatorChar, [System.IO.Path]::AltDirectorySeparatorChar)
$normalizedFile = $fileFullPath.Replace([System.IO.Path]::AltDirectorySeparatorChar, [System.IO.Path]::DirectorySeparatorChar)
$comparison = if ($IsWindows) {
[System.StringComparison]::OrdinalIgnoreCase
}
else {
[System.StringComparison]::Ordinal
}
$workspacePrefix = $normalizedWorkspace + [System.IO.Path]::DirectorySeparatorChar
if ($normalizedFile.StartsWith($workspacePrefix, $comparison) -or $normalizedFile.Equals($normalizedWorkspace, $comparison)) {
$relativePath = if ($normalizedFile.Equals($normalizedWorkspace, $comparison)) {
''
}
else {
$normalizedFile.Substring($normalizedWorkspace.Length + 1)
}
if ($relativePath) {
$filePath = $relativePath
}
}
}
# Normalize to forward slashes for consistency (required by GitHub annotations)
$filePath = $filePath -replace '\\', '/'

Copilot uses AI. Check for mistakes.
# Create annotation title
$annotationTitle = "Test Failure: $description / $testName"

# Build the annotation message
$annotationMessage = $message -replace "`n", "%0A" -replace "`r"

Copilot AI Feb 12, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$annotationMessage = $message -replace "n", "%0A" -replace "r" is missing the replacement value for the second -replace, which will cause a PowerShell parse error at runtime. Provide an explicit replacement (e.g., strip \r or encode it) so the action can run.

Suggested change
$annotationMessage = $message -replace "`n", "%0A" -replace "`r"
$annotationMessage = $message -replace "`n", "%0A" -replace "`r", "%0D"

Copilot uses AI. Check for mistakes.

# Build and output the workflow command
$workflowCommand = "::error file=$filePath"
if ($fileInfo.Line) {
$workflowCommand += ",line=$($fileInfo.Line)"
}
$workflowCommand += ",title=$annotationTitle::$annotationMessage"
Comment on lines +79 to +83

Copilot AI Feb 12, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GitHub Actions workflow command needs escaping for %, \r, \n, and for property values also : and ,. As written, $filePath, $annotationTitle, or $annotationMessage containing these characters can produce a malformed ::error ...::... command (or truncate fields). Add proper workflow-command escaping before concatenating file=, title=, and the message payload.

Copilot uses AI. Check for mistakes.

Write-Host $workflowCommand

# Output a link to the test run
if ($env:GITHUB_SERVER_URL -and $env:GITHUB_REPOSITORY -and $env:GITHUB_RUN_ID) {
$logUrl = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)"
Write-Host "Test logs: $logUrl"
}
}
}
}

@"
Expand Down
63 changes: 63 additions & 0 deletions build.psm1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1864,6 +1864,69 @@ $stack_trace

}

function Get-PesterFailureFileInfo
{
[CmdletBinding()]
param (
[Parameter(Mandatory)]
[string]$StackTraceString
)

# Parse stack trace to extract file path and line number
# Common patterns:
# "at line: 123 in C:\path\to\file.ps1" (Pester 4)
# "at C:\path\to\file.ps1:123"
# "at <ScriptBlock>, C:\path\to\file.ps1: line 123"
# "at 1 | Should -Be 2, /path/to/file.ps1:123" (Pester 5)
# "at 1 | Should -Be 2, C:\path\to\file.ps1:123" (Pester 5 Windows)

$result = @{
File = $null
Line = $null
}

if ([string]::IsNullOrWhiteSpace($StackTraceString)) {
return $result
}

# Try pattern: "at line: 123 in <path>" (Pester 4)
if ($StackTraceString -match 'at line:\s*(\d+)\s+in\s+(.+?)(?:\r|\n|$)') {
$result.Line = $matches[1]
$result.File = $matches[2].Trim()
return $result
}

# Try pattern: ", <path>:123" (Pester 5 format)
# This handles both Unix paths (/path/file.ps1:123) and Windows paths (C:\path\file.ps1:123)
if ($StackTraceString -match ',\s*((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1):(\d+)') {
$result.File = $matches[1].Trim()
$result.Line = $matches[2]
return $result
}

# Try pattern: "at <path>:123" (without comma)
# Handle both absolute Unix and Windows paths
if ($StackTraceString -match 'at\s+((?:[A-Za-z]:)?[\/\\][^,]+?\.ps[m]?1):(\d+)(?:\r|\n|$)') {
$result.File = $matches[1].Trim()
$result.Line = $matches[2]
return $result
}

# Try pattern: "<path>: line 123"
if ($StackTraceString -match '((?:[A-Za-z]:)?[\/\\][^,]+?\.ps[m]?1):\s*line\s+(\d+)(?:\r|\n|$)') {
$result.File = $matches[1].Trim()
$result.Line = $matches[2]
return $result
}

# Try to extract just the file path if no line number found
if ($StackTraceString -match '(?:at\s+|in\s+)?((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1)') {
$result.File = $matches[1].Trim()
}

return $result
}

function Test-XUnitTestResults
{
param(
Expand Down
Loading