Adds more travis test targets#3039
Conversation
There was a problem hiding this comment.
from the travis CI documentation:
we encourage users to specify node and/or iojs to run using the latest stable releases.
so i would keep this to "node"
also we have seen a significant perf degradation with 0.12 and that will make tests take even longer.
There was a problem hiding this comment.
OK I keep one min. node version to 0.10 and one node and iojs latest?
node_js:
- iojs # latest iojs version
- node # latest node
- '0.10' # min supportThere was a problem hiding this comment.
I get the change in the harness, but I don't get the change in the .travis.yml file - why is this necessary?
There was a problem hiding this comment.
@tinganho stupid question, does it run the tree targets all the time? i.e. it would not considered passed until the test passed on the three targets? if so i would keep it to two if we can.
There was a problem hiding this comment.
@mhegazy yes it will run all environments every time. Though some times not starting at the same time. All environments must pass to be considered a pass.
Keep it to iojs and node?
There was a problem hiding this comment.
@DanielRosenwasser I just think it is common practice to test multiple environments. Some test fails sometimes on different environments but not others.
|
How would this change affect our test time? any estimates? |
|
@danquirk can you take a look? |
|
@mhegazy if vm:s avail. they run the same time. |
|
I don't think this is worth the slowdown. Unless these new test targets are running in parallel both locally and on Travis this is going to add significant overhead to test run time for little gain. How often are we going to see an issue on node and not io or vice versa? It hasn't happened so far that I can recall (or no one has reported it). We could choose to run against a variety of versions of node, io and chakra on each checkin for maximum coverage, but we've determined the slowdown isn't worth the miniscule overall increase in correctness coverage. Perhaps in the future if these platforms diverge more the value of targeting both all the time increases to a point where we ant this. You can see the Travis run details from this PR as an example of the perf changes:
|
|
based on @danquirk i would say let's not do that. |
|
OK, but probably you should add a fix to harness. I couldn't run the test suit in iojs. But besides from that I agree on @danquirk point. Btw, how do you guys develop using the test? Change or add some thing and then run the test(with |
|
You can run individual tests or sets of tests by passing the t= or tests= parameter to runtests. It just forwards that value to mocha's grep command grabbing all tests with the relevant text in their describe block. All compiler and fourslash tests have the relative filepath in that block so you can do things like:
Also you can use the r= or reporter= command to change how the results are displayed which is particularly useful with t=. So for instance you might want to do:
If you want to submit a separate PR with just the io.js related fix that'd be appreciated. |
|
Just submitted a separate PR with the node check #3050 |
This PR adds more Travis test targets. The
iojstargets defaults to latest stable. There is also a fix issue with the node check environmentprocess.execPathhas the path**/iojsinstead of**/nodein iojs.