Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle tests marked as inconclusive #2405

Merged
merged 19 commits into from
Apr 10, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8e1413e
Adds handling of It's marked as inconclusive
csandfeld Nov 19, 2023
5ec791c
Removes **DEPRECATED** from Inconclusive parameter help text
csandfeld Nov 19, 2023
7eb17cd
Adds deprecation notice to pester report when Set-ItResult -Pending i…
csandfeld Nov 19, 2023
f5626bf
Fix typo and add changes to fix failing P tests
csandfeld Nov 20, 2023
53acb19
Moves deprecation notice to end of Pester report
csandfeld Jan 19, 2024
79d8327
Fix InconclusiveCount typo
csandfeld Jan 19, 2024
6710d33
Adds missing increment of OwnInconclusiveCount
csandfeld Jan 19, 2024
b9237e8
Increment InconclusiveCount in Pester4Result object
csandfeld Jan 19, 2024
7c219c2
Update example in Set-ItResult comment based help
csandfeld Feb 3, 2024
9ba12d5
Test for Inconclusive tests and count in ResultObject
csandfeld Feb 3, 2024
a21a550
Adds inconclusive and skipped tests to nunit report 'ts' files
csandfeld Feb 3, 2024
ce3da79
Suppress output and use consistent configuration style
csandfeld Apr 5, 2024
b77ce8a
Adds skipped and inconclusive testcases to schema validation test
csandfeld Apr 5, 2024
82e93f3
Refactors nunit reports inconclusive count and skipped count tests
csandfeld Apr 5, 2024
6cf60c6
Removes unnecessary assertions
csandfeld Apr 5, 2024
066ca5d
Set Inconclusive result in NUnit reports
csandfeld Apr 7, 2024
9d80eb5
Add cheaper check for deprecation, and a test, remove INCONCLUSIVE fr…
nohwnd Apr 10, 2024
c156209
Format because
nohwnd Apr 10, 2024
d5a5e1a
Fix pending detection on failing run
nohwnd Apr 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions src/Main.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -747,12 +747,12 @@ function Invoke-Pester {
}

$plugins.Add((
# decorator plugin needs to be added after output
# because on teardown they will run in opposite order
# and that way output can consume the fixed object that decorator
# decorated, not nice but works
Get-RSpecObjectDecoratorPlugin
))
# decorator plugin needs to be added after output
# because on teardown they will run in opposite order
# and that way output can consume the fixed object that decorator
# decorated, not nice but works
Get-RSpecObjectDecoratorPlugin
))

if ($PesterPreference.TestDrive.Enabled.Value) {
$plugins.Add((Get-TestDrivePlugin))
Expand Down Expand Up @@ -1458,11 +1458,13 @@ function ConvertTo-Pester4Result {
"Skipped" {
$legacyResult.SkippedCount++
}
"Inconclusive" {
$legacyResult.InconclusiveCount++
}
}
}
$legacyResult.TotalCount = $legacyResult.TestResult.Count
$legacyResult.PendingCount = 0
$legacyResult.InconclusiveCount = 0
$legacyResult.Time = $PesterResult.Duration

$legacyResult
Expand Down
5 changes: 4 additions & 1 deletion src/Pester.Runtime.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -2211,6 +2211,9 @@ function PostProcess-ExecutedBlock {
if (-not $t.ShouldRun) {
$b.OwnNotRunCount++
}
elseif ($t.ShouldRun -and $t.Inconclusive) {
$b.OwnInconclusiveCount++
}
elseif ($t.ShouldRun -and $t.Skipped) {
$b.OwnSkippedCount++
}
Expand Down Expand Up @@ -2273,7 +2276,7 @@ function PostProcess-ExecutedBlock {
$b.PassedCount += $child.PassedCount
$b.FailedCount += $child.FailedCount
$b.SkippedCount += $child.SkippedCount
$b.InconclusiveCount += $child.OwnInconclusiveCount
$b.InconclusiveCount += $child.InconclusiveCount
$b.NotRunCount += $child.NotRunCount
}

Expand Down
9 changes: 5 additions & 4 deletions src/functions/Output.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -310,10 +310,6 @@ function Write-PesterReport {
)
# if(-not ($PesterState.Show | Has-Flag Summary)) { return }

if ($RunResult.Tests.ErrorRecord.FullyQualifiedErrorID -contains 'PesterTestPending') {
Write-PesterHostMessage '**DEPRECATED**: The Set-ItResult -Pending parameter is deprecated. It will be removed in a future version of Pester.' -ForegroundColor $ReportTheme.Warning
}

Write-PesterHostMessage ($ReportStrings.Timing -f (Get-HumanTime ($RunResult.Duration))) -Foreground $ReportTheme.Foreground

$Success, $Failure = if ($RunResult.FailedCount -gt 0) {
Expand Down Expand Up @@ -408,6 +404,11 @@ function Write-PesterReport {
# & $SafeCommands['Write-Host'] ($ReportStrings.TestsPending -f $RunResult.PendingCount) -Foreground $Pending -NoNewLine
# & $SafeCommands['Write-Host'] ($ReportStrings.TestsInconclusive -f $RunResult.InconclusiveCount) -Foreground $Inconclusive
# }

if ($RunResult.Tests.ErrorRecord.FullyQualifiedErrorID -contains 'PesterTestPending') {
Write-PesterHostMessage ''
Write-PesterHostMessage '**DEPRECATED**: The Set-ItResult -Pending parameter is deprecated. It will be removed in a future version of Pester.' -ForegroundColor $ReportTheme.Warning
}
}

function Write-CoverageReport {
Expand Down
11 changes: 8 additions & 3 deletions src/functions/Set-ItResult.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@
.EXAMPLE
```powershell
Describe "Example" {
It "Inconclusive test" {
Set-ItResult -Inconclusive -Because "we want it to be inconclusice"
}
It "Skipped test" {
Set-ItResult -Skipped -Because "we want it to be skipped"
}
Expand All @@ -38,9 +41,11 @@
the output should be

```
[!] Skipped test is skipped, because we want it to be skipped
Tests completed in 0ms
Tests Passed: 0, Failed: 0, Skipped: 0, Pending: 0, Inconclusive 1
Describing Example
[?] Inconclusive test is inconclusive, because INCONCLUSIVE: we want it to be inconclusice 35ms (32ms|3ms)
[!] Skipped test is skipped, because we want it to be skipped 3ms (2ms|1ms)
Tests completed in 78ms
Tests Passed: 0, Failed: 0, Skipped: 1, Inconclusive: 1, NotRun: 0
```

.LINK
Expand Down
15 changes: 13 additions & 2 deletions tst/Pester.RSpec.ResultObject.ts.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,14 @@ i -PassThru:$PassThru {
It "not run" -Tag "Slow" {
1 | Should -Be 1
}

It "Set-ItResult -Inconclusive" {
Set-ItResult -Inconclusive
}

It "Set-ItResult -Skipped" {
Set-ItResult -Skipped
}
}
}
$result = Invoke-Pester -Configuration @{
Expand All @@ -119,7 +127,7 @@ i -PassThru:$PassThru {
$result | Verify-Property "Containers"
$result.Containers.Count | Verify-Equal 2

$result.TotalCount | Verify-Equal 4
$result.TotalCount | Verify-Equal 6
$result.Tests | Verify-NotNull

$result.PassedCount | Verify-Equal 1
Expand All @@ -128,12 +136,15 @@ i -PassThru:$PassThru {
$result.FailedCount | Verify-Equal 1
$result.Failed | Verify-NotNull

$result.SkippedCount | Verify-Equal 1
$result.SkippedCount | Verify-Equal 2
$result.Skipped | Verify-NotNull

$result.NotRunCount | Verify-Equal 1
$result.NotRun | Verify-NotNull

$result.InconclusiveCount | Verify-Equal 1
$result.Inconclusive | Verify-NotNull

$result.Duration | Verify-Equal ($result.Containers[0].Duration + $result.Containers[1].Duration)
$result.UserDuration | Verify-Equal ($result.Containers[0].UserDuration + $result.Containers[1].UserDuration)
$result.FrameworkDuration | Verify-Equal ($result.Containers[0].FrameworkDuration + $result.Containers[1].FrameworkDuration)
Expand Down
98 changes: 98 additions & 0 deletions tst/Pester.RSpec.TestResults.NUnit25.ts.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,51 @@ i -PassThru:$PassThru {

}

t "should write a skipped test result" {
$sb = {
Describe "Mocked Describe 1" {
It "Skipped testcase" -Skip {
}
}
Describe "Mocked Describe 2" {
It "Skipped testcase" {
Set-ItResult -Skipped
}
}
}
$r = Invoke-Pester -Configuration ([PesterConfiguration]@{ Run = @{ ScriptBlock = $sb; PassThru = $true }; Output = @{ Verbosity = 'None' } })

$xmlResult = $r | ConvertTo-NUnitReport
$xmlTestSuite = $xmlResult.'test-results'.'test-suite'.'results'.'test-suite'.'results'.'test-suite'
$xmlTestCase1 = $xmlTestSuite.results.'test-case'[0]
$xmlTestCase2 = $xmlTestSuite.results.'test-case'[1]

$xmlTestCase1.name | Verify-Equal "Mocked Describe 1.Skipped testcase"
$xmlTestCase1.result | Verify-Equal "Ignored"
$xmlTestCase1.time | Verify-XmlTime $r.Containers[0].Blocks[0].Tests[0].Duration

$xmlTestCase2.name | Verify-Equal "Mocked Describe 2.Skipped testcase"
$xmlTestCase2.result | Verify-Equal "Ignored"
$xmlTestCase2.time | Verify-XmlTime $r.Containers[0].Blocks[1].Tests[0].Duration
}

t "should write an inconclusive test result" {
$sb = {
Describe "Mocked Describe" {
It "Inconclusive testcase" {
Set-ItResult -Inconclusive
}
}
}
$r = Invoke-Pester -Configuration ([PesterConfiguration]@{ Run = @{ ScriptBlock = $sb; PassThru = $true }; Output = @{ Verbosity = 'None' } })

$xmlResult = $r | ConvertTo-NUnitReport
$xmlTestCase = $xmlResult.'test-results'.'test-suite'.'results'.'test-suite'.'results'.'test-suite'.'results'.'test-case'
$xmlTestCase.name | Verify-Equal "Mocked Describe.Inconclusive testcase"
$xmlTestCase.result | Verify-Equal "Inconclusive"
$xmlTestCase.time | Verify-XmlTime $r.Containers[0].Blocks[0].Tests[0].Duration
}

t "should write the test summary" {
$sb = {
Describe "Mocked Describe" {
Expand All @@ -162,6 +207,59 @@ i -PassThru:$PassThru {
$xmlTestResult.time | Verify-Equal (Get-Date -Format "HH:mm:ss" $r.ExecutedAt)
}

t "should write inconclusive count" {
$sb = {
Describe "Mocked Describe" {
It "Inconclusive testcase" {
Set-ItResult -Inconclusive
}
}
}
$r = invoke-pester -container ( new-pestercontainer -ScriptBlock $sb) -passThru -Output Detailed
$xmlResult = $r | ConvertTo-NUnitReport
$xmlResult.'test-results'.inconclusive | Verify-Equal 1

fflaten marked this conversation as resolved.
Show resolved Hide resolved
$sb = {
Describe "Mocked Describe" {
It "Inconclusive testcase 1" {
Set-ItResult -Inconclusive
}
It "Inconclusive testcase 2" {
Set-ItResult -Inconclusive
}
}
}
fflaten marked this conversation as resolved.
Show resolved Hide resolved
$r = invoke-pester -container ( new-pestercontainer -ScriptBlock $sb) -passThru -Output Detailed
fflaten marked this conversation as resolved.
Show resolved Hide resolved
$xmlResult = $r | ConvertTo-NUnitReport
$xmlResult.'test-results'.inconclusive | Verify-Equal 2
}

t "should write skipped count" {
$sb = {
Describe "Mocked Describe" {
It "Skipped testcase" {
Set-ItResult -Skipped
}
}
}
$r = invoke-pester -container ( new-pestercontainer -ScriptBlock $sb) -passThru -Output Detailed
$xmlResult = $r | ConvertTo-NUnitReport
$xmlResult.'test-results'.skipped | Verify-Equal 1

$sb = {
Describe "Mocked Describe" {
It "Skipped testcase 1" -Skip {
}
It "Skippde testcase 2" {
Set-ItResult -Skipped
}
}
}
$r = invoke-pester -container ( new-pestercontainer -ScriptBlock $sb) -passThru -Output Detailed
$xmlResult = $r | ConvertTo-NUnitReport
$xmlResult.'test-results'.skipped | Verify-Equal 2
fflaten marked this conversation as resolved.
Show resolved Hide resolved
}

t "should write the test-suite information" {
$sb = {
Describe "Mocked Describe" {
Expand Down
Loading
Loading