Skip to content

Commit 08b13f2

Browse files
TravisEz13TravisEz13
authored
Add Pester CI Analysis Skill (#26806)
1 parent 6b8d724 commit 08b13f2

File tree

5 files changed

+1445
-0
lines changed

5 files changed

+1445
-0
lines changed

‎.github/instructions/pester-test-status-and-working-meaning.instructions.md‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
---
2+
applyTo: "**/*.Tests.ps1"
3+
---
4+
5+
# Pester Test Status Meanings and Working Tests
6+
7+
## Purpose
8+
9+
This guide clarifies Pester test outcomes and what it means for a test to be "working" - which requires both **passing** AND **actually validating functionality**.
10+
11+
## Test Statuses in Pester
12+
13+
### Passed ✓
14+
**Status Code**: `Passed`
15+
**Exit Result**: Test ran successfully, all assertions passed
16+
17+
**What it means**:
18+
- Test executed without errors
19+
- All `Should` statements evaluated to true
20+
- Test setup and teardown completed without issues
21+
- Test is **validating** the intended functionality
22+
23+
**What it does NOT mean**:
24+
- The feature is working (assertions could be wrong)
25+
- The test is meaningful (could be testing wrong thing)
26+
- The test exercises all code paths
27+
28+
### Failed ✗
29+
**Status Code**: `Failed`
30+
**Exit Result**: Test ran but assertions failed
31+
32+
**What it means**:
33+
- Test executed but an assertion returned false
34+
- Expected value did not match actual value
35+
- Test detected a problem with the functionality
36+
37+
**Examples**:
38+
```
39+
Expected $true but got $false
40+
Expected 5 items but got 3
41+
Expected no error but got: Cannot find parameter
42+
```
43+
44+
### Error ⚠
45+
**Status Code**: `Error`
46+
**Exit Result**: Test crashed with an exception
47+
48+
**What it means**:
49+
- Test failed to complete
50+
- An exception was thrown during test execution
51+
- Could be in test setup, test body, or test cleanup
52+
- Often indicates environmental issue, not code functional issue
53+
54+
**Examples**:
55+
```
56+
Cannot bind argument to parameter 'Path' because it is null
57+
File not found: C:\expected\config.json
58+
Access denied writing to registry
59+
```
60+
61+
### Pending ⏳
62+
**Status Code**: `Pending`
63+
**Exit Result**: Test ran but never completed assertions
64+
65+
**What it means**:
66+
- Test was explicitly marked as not ready to run
67+
- `Set-ItResult -Pending` was called
68+
- Used to indicate: known bugs, missing features, environmental issues
69+
70+
**When to use Pending**:
71+
- Test for feature in development
72+
- Test disabled due to known bug (issue #1234)
73+
- Test disabled due to intermittent failures being fixed
74+
- Platform-specific issues being resolved
75+
76+
**⚠️ WARNING**: Pending tests are NOT validating functionality. They hide problems.
77+
78+
### Skipped ⊘
79+
**Status Code**: `Skipped`
80+
**Exit Result**: Test did not run (detected at start)
81+
82+
**What it means**:
83+
- Test was intentionally not executed
84+
- `-Skip` parameter or `It -Skip:$condition` was used
85+
- Environment doesn't support this test
86+
87+
**When to use Skip**:
88+
- Test not applicable to current platform (Windows-only test on Linux)
89+
- Test requires feature that's not available (admin privileges)
90+
- Test requires specific configuration not present
91+
92+
**Difference from Pending**:
93+
- **Skip**: "This test shouldn't run here" (known upfront)
94+
- **Pending**: "This test should eventually run but can't now"
95+
96+
### Ignored ✛
97+
**Status Code**: `Ignored`
98+
**Exit Result**: Test marked as not applicable
99+
100+
**What it means**:
101+
- Test has `[Ignore("reason")]` attribute
102+
- Test is permanently disabled in this location
103+
- Not the same as Skipped (which is conditional)
104+
105+
**When to use Ignore**:
106+
- Test for deprecated feature
107+
- Test for bug that won't be fixed
108+
- Test moved to different test file
109+
110+
---
111+
112+
## What Does "Working" Actually Mean?
113+
114+
A test is **working** when it meets BOTH criteria:
115+
116+
### 1. **Test Status is PASSED** ✓
117+
```powershell
118+
It "Test name" {
119+
# Test executes
120+
# All assertions pass
121+
# Returns Passed status
122+
}
123+
```
124+
125+
### 2. **Test Actually Validates Functionality**
126+
```powershell
127+
# ✓ GOOD: Tests actual functionality
128+
It "Get-Item returns files from directory" -Tags @('Unit') {
129+
$testDir = New-Item -ItemType Directory -Force
130+
New-Item -Path $testDir -Name "file.txt" -ItemType File | Out-Null
131+
132+
$result = Get-Item -Path "$testDir\file.txt"
133+
134+
$result.Name | Should -Be "file.txt"
135+
$result | Should -Exist
136+
137+
Remove-Item $testDir -Recurse -Force
138+
}
139+
140+
# ✗ BAD: Returns Passed but doesn't validate functionality
141+
It "Get-Item returns files from directory" -Tags @('Unit') {
142+
$result = Get-Item -Path somepath # May not exist, may not actually test
143+
$result | Should -Not -BeNullOrEmpty # Too vague
144+
}
145+
146+
# ✗ BAD: Test marked Pending - validation is hidden
147+
It "Get-Item returns files from directory" -Tags @('Unit') {
148+
Set-ItResult -Pending -Because "File system not working"
149+
return
150+
# No validation happens at all
151+
}
152+
```
153+
154+
---
155+
156+
## The Problem with Pending Tests
157+
158+
### Why Pending Tests Hide Problems
159+
160+
```powershell
161+
# BAD: Test marked Pending - looks like "working" status but validation is skipped
162+
It "Download help from web" {
163+
Set-ItResult -Pending -Because "Web connectivity issues"
164+
return
165+
166+
# This code never runs:
167+
Update-Help -Module PackageManagement -Force -ErrorAction Stop
168+
Get-Help Get-Package | Should -Not -BeNullOrEmpty
169+
}
170+
```
171+
172+
**Result**:
173+
- ✗ Feature is broken (Update-Help fails)
174+
- ✓ Test shows "Pending" (looks acceptable)
175+
- ✗ Problem is hidden and never fixed
176+
177+
### The Right Approach
178+
179+
**Option A: Fix the root cause**
180+
```powershell
181+
It "Download help from web" {
182+
# Use local assets that are guaranteed to work
183+
Update-Help -Module PackageManagement -SourcePath ./assets -Force -ErrorAction Stop
184+
185+
Get-Help Get-Package | Should -Not -BeNullOrEmpty
186+
}
187+
```
188+
189+
**Option B: Gracefully skip when unavailable**
190+
```powershell
191+
It "Download help from web" -Skip:$(-not $hasInternet) {
192+
Update-Help -Module PackageManagement -Force -ErrorAction Stop
193+
Get-Help Get-Package | Should -Not -BeNullOrEmpty
194+
}
195+
```
196+
197+
**Option C: Add retry logic for intermittent issues**
198+
```powershell
199+
It "Download help from web" {
200+
$maxRetries = 3
201+
$attempt = 0
202+
203+
while ($attempt -lt $maxRetries) {
204+
try {
205+
Update-Help -Module PackageManagement -Force -ErrorAction Stop
206+
break
207+
}
208+
catch {
209+
$attempt++
210+
if ($attempt -ge $maxRetries) { throw }
211+
Start-Sleep -Seconds 2
212+
}
213+
}
214+
215+
Get-Help Get-Package | Should -Not -BeNullOrEmpty
216+
}
217+
```
218+
219+
---
220+
221+
## Test Status Summary Table
222+
223+
| Status | Passed? | Validates? | Counts as "Working"? | Use When |
224+
|--------|---------|------------|----------------------|----------|
225+
| **Passed** | ✓ | ✓ | **YES** | Feature is working and test proves it |
226+
| **Failed** | ✗ | ✓ | NO | Feature is broken or test has wrong expectation |
227+
| **Error** | ✗ | ✗ | NO | Test infrastructure broken, can't validate |
228+
| **Pending** | - | ✗ | **NO** ⚠️ | Temporary - test should eventually pass |
229+
| **Skipped** | - | ✗ | NO | Test not applicable to this environment |
230+
| **Ignored** | - | ✗ | NO | Test permanently disabled |
231+
232+
---
233+
234+
## Recommended Patterns
235+
236+
### Pattern 1: Resilient Test with Fallback
237+
```powershell
238+
It "Feature works with web or local source" {
239+
$useLocal = $false
240+
241+
try {
242+
Update-Help -Module Package -Force -ErrorAction Stop
243+
}
244+
catch {
245+
$useLocal = $true
246+
Update-Help -Module Package -SourcePath ./assets -Force -ErrorAction Stop
247+
}
248+
249+
# Validate functionality regardless of source
250+
Get-Help Get-Package | Should -Not -BeNullOrEmpty
251+
}
252+
```
253+
254+
### Pattern 2: Conditional Skip with Clear Reason
255+
```powershell
256+
Describe "Update-Help from Web" -Skip $(-not (Test-InternetConnectivity)) {
257+
It "Downloads help successfully" {
258+
Update-Help -Module PackageManagement -Force -ErrorAction Stop
259+
Get-Help Get-Package | Should -Not -BeNullOrEmpty
260+
}
261+
}
262+
```
263+
264+
### Pattern 3: Separate Suites by Dependency
265+
```powershell
266+
Describe "Help Content Tests - Web" {
267+
# Tests that require internet - can be skipped if unavailable
268+
It "Downloads from web" { ... }
269+
}
270+
271+
Describe "Help Content Tests - Local" {
272+
# Tests with local assets - should always pass
273+
It "Loads from local assets" {
274+
Update-Help -Module Package -SourcePath ./assets -Force
275+
Get-Help Get-Package | Should -Not -BeNullOrEmpty
276+
}
277+
}
278+
```
279+
280+
---
281+
282+
## Checklist: Is Your Test "Working"?
283+
284+
- [ ] Test status is **Passed** (not Pending, not Skipped, not Failed)
285+
- [ ] Test actually **executes** the feature being tested
286+
- [ ] Test has **specific assertions** (not just `Should -Not -BeNullOrEmpty`)
287+
- [ ] Test includes **cleanup** (removes temp files, restores state)
288+
- [ ] Test can run **multiple times** without side effects
289+
- [ ] Test failure **indicates a real problem** (not flaky assertions)
290+
- [ ] Test success **proves the feature works** (not just "didn't crash")
291+
292+
If any of these is false, your test may be passing but not "working" properly.
293+
294+
---
295+
296+
## See Also
297+
298+
- [Pester Documentation](https://pester.dev/)
299+
- [Set-ItResult Documentation](https://pester.dev/docs/commands/Set-ItResult)

0 commit comments

Comments
 (0)