Jameswtruher/travisdailybuild by JamesWTruher · Pull Request #2958 · PowerShell/PowerShell

JamesWTruher · 2017-01-05T22:25:18Z

enable daily test run support in travis

This also includes a number of test changes which need to be reviewed by the test owners
@lzybkr I've made changes in parser tests to support async execution to avoid test run hangs which have sporadically appeared. Now, if one of those tests hang, it (the test) will fail rather than the entire run.
@Francisco-Gamino, I've made changes to the help tests, counter and management tests

once this is merged, I'll have an addition PR to put the test run badge in the ReadMe.md (after I create the cron job to actually run the tests)

Modify test failure presentation to use platform available XML methods

Some of the language/parser tests have been hanging in a non-reproducable manner which causes the CI system to invalidate the entire run. This change adds support for timeout which will fail a test if it runs to long, rather than invalidate the entire run. current behavior is still supported, and is not done in a new session: PS> get-runtimeerror -src '1/' At line:1 char:3 + 1/ + ~ You must provide a value expression following the '/' operator. + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : ExpectedValueExpression Adding a timeout will do the operation in a async powershell session PS> get-runtimeerror -src '1/' -timeout 5 You must provide a value expression following the '/' operator. + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : ExpectedValueExpression If the operation takes longer than the supplied timeout, a timeout error will be returned PS> get-runtimeerror -src 'start-sleep 6' -timeout 2 get-runtimeerror : Operation Timed Out ('start-sleep 6') At line:1 char:1 + get-runtimeerror -src 'start-sleep 6' -timeout 2 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Get-RuntimeError

Also, only call add-type in CounterTestHelperFunctions.ps1 if we're going to actually run the tests

Travis watches output from the build to ensure that it hasn't hung we need to find a balance between too much output and not enough output. A run which has too much output is killed because it looks like an error loop A run which has too little output is killed because it looks like a hang

adityapatwardhan · 2017-01-05T22:41:38Z


        $cmd = ConstructCommand $testCase
-        Write-Host "Command to run: $cmd"
+        # Write-Host "Command to run: $cmd"


Should just delete this line. #Resolved

done! #Closed

adityapatwardhan · 2017-01-05T22:42:43Z

    Get-InstalledModule -Name $ContosoServer -AllVersions -ErrorAction SilentlyContinue | Uninstall-Module -Force
 }

+


Extra line. #Resolved

deleted #Closed

adityapatwardhan · 2017-01-05T22:45:18Z

    log ("Name='{0}', Destination='{1}', Repository='{2}'" -f ($Name -join ','), $Destination, $RepositoryName)

+    # do not output progress
+    $ProgressPreference = "SilentlyContinue"


Do we need to reset this back to previous setting when we complete the script? #ByDesign

nope - test tests are executed in a new scope, so it gets set back when the script exits #Closed

Remove extraneous extra line in PowerShellGet.Tests.ps1

lzybkr

Help me understand the timeout on the parser tests - the test will still fail, but just fail faster? Or something else?

I wonder how hard it would be to get a core dump when we see the timeout - upload that as an artifact, and debug later.

lzybkr · 2017-01-06T00:00:58Z

    }
    catch 
    {
+        write-verbose -verbose "caught"


Is this a left-over debugging message?
caught isn't a very useful message by itself, and I don't see any other Write-Verbose calls in this function to give it context. #Resolved

indeed yes - pulled

In reply to: 94878328 [](ancestors = 94878328)

lzybkr · 2017-01-06T00:05:15Z

        {
            It "error should happen at parse time, not at runtime" -Skip {}
-            $errors = Get-RuntimeError -Src $src
+            $errors = Get-RuntimeError -Src $src -Timeout 5


I'd feel a bit better if we waited a bit longer, maybe 10 seconds - in case we're on a really slow VM or we add a test that is somewhat longish (like 1s or so). #Resolved

it was SWAG for sure, I don't have trouble with having this be 10

In reply to: 94878763 [](ancestors = 94878763)

lzybkr · 2017-01-06T00:13:40Z

        logerror "TEST FAILURES"
-        foreach ( $testfail in $x.SelectNodes('.//test-case[@result = "Failure"]'))
+        # switch between methods, SelectNode is not available on dotnet core
+        if ( "System.Xml.XmlDocumentXPathExtensions" -as "Type" ) {


As a general rule, you should prefer -as [Type] instead of -as "Type" because the former resolves the type just once while the later resolves it every time it is executed. #Resolved

changed

In reply to: 94879724 [](ancestors = 94879724)

iSazonov · 2017-01-06T11:15:39Z


    $script:counterSamples = $null
-    if ($maxSamples)
+    if ($maxSamples -and $IsWindows)


Seems Get-Counter don't work on IoT.
SetScriptVars is only called from BeforeAll blocks so we can mask it there by SkipCounterTests. #Resolved

iSazonov · 2017-01-06T11:16:43Z

    }

-    if ($export)
+    if ($export -and $IsWindows)


The same about IoT. #Resolved

iSazonov · 2017-01-06T15:59:59Z

+            $ps.AddScript($src) > $null
+            $ar = $ps.BeginInvoke()
+            # give it 250 milliseconds to complete
+            start-sleep -mill 250


Parser tests is very fast (some ms). This change will slow them down in many times.
And how will it help us to find the root of the problem? Travis already has a global monitoring of a job duration and automatically kills a hung job. Maybe we ask them to take a dump in this moment? #WontFix

yes, it will slow down some tests, but I reckoned that it was better to slow down the test than to invalidate an entire run - that's what I'm trying to avoid. When running a full test pass I saw nearly 100% test runs killed for lack of activity. Each one of those runs were hung in the language tests.

In reply to: 94967709 [](ancestors = 94967709)

I am concerned that we conceal the problem. Is there a way to reproduce this problem except to get a hung accidentally? I believe that we in the allowed time could identify the problem and fix it by analyzing a dump without slow down tests. #WontFix

while I agree that we need to determine root cause, i disagree about where. Catching this in CI has been extremely difficult and collecting dumps in Travis is really something I would rather avoid (if it's even possible).
we need to track down this issue, but I don't believe CI is the appropriate place

In reply to: 94993680 [](ancestors = 94993680)

There seems to be an issue that needs investigation. Please open an issue for this. #Resolved

an issue has already been created for the hanging behavior. #2748

In reply to: 95024875 [](ancestors = 95024875)

Alter timeout to 10 seconds to be improve chances of not timing out for runtime parser checks improve logic for counter tests to also skip for IoT

iSazonov · 2017-01-06T19:22:01Z

+    $ShouldRun = $false
+}
+
+if ( $ShouldRun )


$ShouldRun looks like duplication of SkipCounterTests #Resolved

Agreed, This is duplicate of SkipCounterTests #Resolved

TravisEz13

~~I'll merge. I expect the comment to be addressed.~~

TravisEz13 · 2017-01-06T21:37:07Z

+            $ps.AddScript($src) > $null
+            $ar = $ps.BeginInvoke()
+            # give it 250 milliseconds to complete
+            start-sleep -mill 250


There seems to be an issue that needs investigation. Please open an issue for this. #Resolved

TravisEz13 · 2017-01-06T21:37:46Z

+    $ShouldRun = $false
+}
+
+if ( $ShouldRun )


Agreed, This is duplicate of SkipCounterTests #Resolved

Francisco-Gamino · 2017-01-06T20:17:33Z

 function SkipCounterTests
 {
    if ([System.Management.Automation.Platform]::IsLinux -or
-        [System.Management.Automation.Platform]::IsOSX)


We should have a single function with all the logic to determine whether the tests should run. #Resolved

JamesWTruher · 2017-01-06T23:50:14Z

@lzybkr I don't seem to be able to follow up on your question about the timeout in place.

Currently, when one of the parser tests hang, the entire test run is killed after about 10 minutes of inactivity. The timeout code enables us to fail a single test and have the rest of the run continue. I agree that capturing a dump at that point would be fantastic, but I don't know how to do that. Can you provide instructions? I'll be very happy to add it.
#Resolved

JamesWTruher · 2017-01-06T23:52:21Z

@TravisEz13 an issue has already been created for the hanging behavior. #2748 #Resolved

lzybkr · 2017-01-07T00:03:46Z

I don't know, but a search of 'create linux core dump' suggests that gcore or kill may be capable of creating the core dump.

iSazonov · 2017-01-07T07:53:37Z

It seems it is impossible that hung process killed himself and unloaded a dump.
As I mentioned above the best way to get the dump is to ask Travis team add the new option to travis.yml They already control session timeout and may at this point easily take a dump and upload it in the specified (in travis.yml) location. This can be useful in the future to solve other problems.

Possible workarround http://jsteemann.github.io/blog/2014/10/30/getting-core-dumps-of-failed-travisci-builds/

lzybkr · 2017-01-07T16:53:59Z

We don't know if the process is hung, but clearly a thread is hung, so Jim's change may help in creating the dump. But I agree the CI system should provide a way to debug of they don't already. Appveyor definitely does provide rdp access, maybe Travis already has something similar.

iSazonov · 2017-01-07T19:23:32Z

I found nothing in Travis help but I discovered long discussion about travis dumps travis-ci/travis-ci#3754

JamesWTruher · 2017-01-08T22:28:32Z

will that provide you with what you need, given managed code? As you're the author/owner of these particular tests, shall I just open an issue for you to track?

In reply to: 271043086 [](ancestors = 271043086)

…he logic in import-counter.tests.ps1

JamesWTruher · 2017-01-09T19:29:35Z

@Francisco-Gamino, @TravisEz13 my last commit should have addressed your request

lzybkr · 2017-01-25T23:17:20Z

@JamesWTruher

If I'm reading the logs correctly, the async changes didn't help at all, and most likely are a net negative right now. See these logs:

https://travis-ci.org/PowerShell/PowerShell/jobs/195355364

Notice 10 minutes with no output - so we failed to kill the async test and did not get further results.
Also notice the error tests take ~250ms when they should really take about ~1ms or less.

iSazonov · 2017-01-27T13:24:53Z

The same https://travis-ci.org/PowerShell/PowerShell/jobs/195856711

JamesWTruher added 8 commits January 5, 2017 13:20

Stifle progress output in build.psm1 for some operations

3ff1bea

Modify test failure presentation to use platform available XML methods

Modify native linux command tests to skip on Windows and pending on Mac

ae15e5f

remove verbose and progress output from help tests

d495bf8

Be sure that Feature Counter tests only run on Windows

1f26d3b

Also, only call add-type in CounterTestHelperFunctions.ps1 if we're going to actually run the tests

do not run any get-computerinfo tests on non-windows systems

53d3e8e

suppress progress output from PowerShell Get tests

4be7f19

JamesWTruher assigned lzybkr, TravisEz13, Francisco-Gamino and adityapatwardhan Jan 5, 2017

msftclas added the cla-not-required label Jan 5, 2017

adityapatwardhan approved these changes Jan 5, 2017

View reviewed changes

Remove commented line in Import-Counter.Tests.ps1

da6f88f

Remove extraneous extra line in PowerShellGet.Tests.ps1

lzybkr reviewed Jan 6, 2017

View reviewed changes

PowerShellTeam added the Review - Needed The PR is being reviewed label Jan 6, 2017

iSazonov reviewed Jan 6, 2017

View reviewed changes

iSazonov mentioned this pull request Jan 6, 2017

Remove errors in counter tests on non-Windows #2914

Closed

iSazonov reviewed Jan 6, 2017

View reviewed changes

Change -as "type" to -as [type] in build.psm1

99112fd

Alter timeout to 10 seconds to be improve chances of not timing out for runtime parser checks improve logic for counter tests to also skip for IoT

iSazonov reviewed Jan 6, 2017

View reviewed changes

TravisEz13 approved these changes Jan 6, 2017

View reviewed changes

TravisEz13 requested changes Jan 6, 2017

View reviewed changes

Francisco-Gamino suggested changes Jan 6, 2017

View reviewed changes

TravisEz13 mentioned this pull request Jan 7, 2017

Flaky hang on Travis during class tests run #2748

Closed

use the existing function of SkipCounterTests rather than duplicate t…

218415a

…he logic in import-counter.tests.ps1

TravisEz13 approved these changes Jan 10, 2017

View reviewed changes

TravisEz13 merged commit c97ca77 into PowerShell:master Jan 10, 2017

JamesWTruher deleted the jameswtruher/travisdailybuild branch January 13, 2017 22:30

TravisEz13 mentioned this pull request Jan 28, 2017

Change from #2958 cause test regressions (no output) #3069

Closed

iSazonov removed the Review - Needed The PR is being reviewed label Mar 27, 2017

		Get-InstalledModule -Name $ContosoServer -AllVersions -ErrorAction SilentlyContinue \| Uninstall-Module -Force
		}

Conversation

JamesWTruher commented Jan 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adityapatwardhan Jan 5, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adityapatwardhan Jan 5, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adityapatwardhan Jan 5, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lzybkr left a comment

Choose a reason for hiding this comment

Uh oh!

lzybkr Jan 6, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 6, 2017

Choose a reason for hiding this comment

Uh oh!

lzybkr Jan 6, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 6, 2017

Choose a reason for hiding this comment

Uh oh!

lzybkr Jan 6, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 6, 2017

Choose a reason for hiding this comment

Uh oh!

iSazonov Jan 6, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iSazonov Jan 6, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iSazonov Jan 6, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 6, 2017

Choose a reason for hiding this comment

Uh oh!

iSazonov Jan 6, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 6, 2017

Choose a reason for hiding this comment

Uh oh!

TravisEz13 Jan 6, 2017 • edited by JamesWTruher Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JamesWTruher Jan 8, 2017

Choose a reason for hiding this comment

JamesWTruher commented Jan 5, 2017 •

edited

Loading

adityapatwardhan Jan 5, 2017 •

edited by JamesWTruher

Loading

JamesWTruher Jan 5, 2017 •

edited

Loading

adityapatwardhan Jan 5, 2017 •

edited by JamesWTruher

Loading

JamesWTruher Jan 5, 2017 •

edited

Loading

adityapatwardhan Jan 5, 2017 •

edited by JamesWTruher

Loading

JamesWTruher Jan 5, 2017 •

edited

Loading

lzybkr Jan 6, 2017 •

edited by JamesWTruher

Loading

lzybkr Jan 6, 2017 •

edited by JamesWTruher

Loading

lzybkr Jan 6, 2017 •

edited by JamesWTruher

Loading

iSazonov Jan 6, 2017 •

edited by JamesWTruher

Loading

iSazonov Jan 6, 2017 •

edited by JamesWTruher

Loading

iSazonov Jan 6, 2017 •

edited by JamesWTruher

Loading

iSazonov Jan 6, 2017 •

edited by JamesWTruher

Loading

TravisEz13 Jan 6, 2017 •

edited by JamesWTruher

Loading

iSazonov Jan 6, 2017 •

edited by JamesWTruher

Loading

TravisEz13 Jan 6, 2017 •

edited by JamesWTruher

Loading

TravisEz13 left a comment •

edited

Loading

TravisEz13 Jan 6, 2017 •

edited by JamesWTruher

Loading

TravisEz13 Jan 6, 2017 •

edited by JamesWTruher

Loading

Francisco-Gamino Jan 6, 2017 •

edited by JamesWTruher

Loading

JamesWTruher commented Jan 6, 2017 •

edited

Loading

JamesWTruher commented Jan 6, 2017 •

edited

Loading