Skip to content

Commit b49b032

Browse files
HainishHainish
authored
Cleaning up test execution & naming conventions (EFForg#14570)
1 parent bd2fe96 commit b49b032

File tree

21 files changed

+162
-156
lines changed

21 files changed

+162
-156
lines changed

‎.travis.yml‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,31 +9,31 @@ services:
99
matrix:
1010
fast_finish: true
1111
include:
12-
- env: INFO="lint"
12+
- env: TEST="lint"
1313
language: node_js
1414
node_js:
1515
- "node"
16-
- env: INFO="unittests"
16+
- env: TEST="unittests"
1717
language: node_js
1818
node_js: node
1919
- "node"
20-
- env: INFO="rules"
21-
- env: INFO="fetch"
22-
- env: INFO="preloaded"
20+
- env: TEST="validations"
21+
- env: TEST="fetch"
22+
- env: TEST="preloaded"
2323
- addons:
2424
chrome: beta
25-
env: INFO="chrome beta" BROWSER=google-chrome-beta
25+
env: TEST="chrome beta" BROWSER=google-chrome-beta
2626
- addons:
2727
chrome: stable
28-
env: INFO="chrome stable" BROWSER=google-chrome-stable
28+
env: TEST="chrome stable" BROWSER=google-chrome-stable
2929
- addons:
3030
firefox: latest
31-
env: INFO="firefox" BROWSER=firefox
31+
env: TEST="firefox" BROWSER=firefox
3232
- addons:
3333
firefox: latest-esr
34-
env: INFO="firefox esr" BROWSER=firefox
34+
env: TEST="firefox esr" BROWSER=firefox
3535
before_install:
3636
- export DISPLAY=:99.0
3737
- sh -e /etc/init.d/xvfb start
38-
before_script: travis_retry ./setup_travis.sh
39-
script: . ./run_travis.sh
38+
before_script: travis_retry test/setup_travis.sh
39+
script: . test/run_travis.sh

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ Avoid using the left-wildcard (`<target host="*.example.com" />`) unless you int
148148

149149
Instead, prefer listing explicit target hosts and a single rewrite from `"^http:"` to `"^https:"`. This saves you time as a ruleset author because each explicit target host automatically creates an implicit test URL, reducing the need to add your own test URLs. These also make it easier for someone reading the ruleset to figure out which subdomains are covered.
150150

151-
If you know all subdomains of a given domain support HTTPS, go ahead and use a left-wildcard, along with a plain rewrite from `"^http:"` to `"^https:"`. Make sure to add a bunch of test URLs for the more important subdomains.
151+
If you know all subdomains of a given domain support HTTPS, go ahead and use a left-wildcard, along with a plain rewrite from `"^http:"` to `"^https:"`. Make sure to add a bunch of test URLs for the more important subdomains.
152152

153153
#### Edge-Case: Right-Wildcards
154154

@@ -334,7 +334,7 @@ In `utils` we have a tool called `hsts-prune` which removes `targets` from rules
334334
Every new pull request automatically has the `hsts-prune` utility applied to it as part of the continual integration process. If a new PR introduces a `target` which is preloaded, it will fail the CI test suite. See:
335335

336336
- `.travis.yml`
337-
- `test/travis.sh`
337+
- `test/run_travis.sh`
338338

339339
* * *
340340

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ Get the packages you need and install a git hook to run tests before push:
99

1010
bash install-dev-dependencies.sh
1111

12-
Run all the tests:
12+
Run the ruleset validations and browser tests:
1313

1414
bash test.sh
1515

@@ -50,7 +50,7 @@ This is the source tree for HTTPS Everywhere for Firefox and Chrome.
5050

5151
Important directories you might want to know about
5252

53-
53+
5454
chromium/ WebExtension source code (for Firefox & Chromium/chrome)
5555
chromium/external External dependencies
5656
chromium/test Unit tests
@@ -60,7 +60,7 @@ Important directories you might want to know about
6060
src/chrome/content/rules Ruleset files live here
6161

6262
test/ Travis unit test source code live here
63-
63+
6464
utils/ Various utilities (includes some Travis test source)
6565

6666
Hacking on the Source Code

‎fetch-test.sh‎

Lines changed: 0 additions & 2 deletions
This file was deleted.

‎ruleset-testing.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,10 @@ Submitting changes to any ruleset that does not meet the coverage requirements
5353
will break the build. This means that even fixes of existing rules may require
5454
additional work to bring them up to snuff.
5555

56-
To run the tests locally, you'll need the https-everywhere-checker, which is now
57-
a submodule of https-everywhere. To set it up, run:
56+
To run the tests locally, first install the development dependencies:
5857

5958
./install-dev-dependencies.sh
6059

6160
To test a specific ruleset:
6261

63-
./fetch-test.sh rules/Example.xml
62+
test/manual.sh rules/Example.xml

‎run_travis.sh‎

Lines changed: 0 additions & 56 deletions
This file was deleted.

‎test.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,6 @@ else
1414
git rev-parse && cd "$(git rev-parse --show-toplevel)"
1515
fi
1616

17-
./test/rules.sh
17+
./test/validations.sh
1818
./test/firefox.sh $@
1919
./test/chromium.sh $@

‎test/manual.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
# This script assists ruleset creators in ensuring that there is proper test
3+
# coverage for their rules, and performs a network "fetch" test to alert the
4+
# contributor of potential problems.
5+
6+
exec python2.7 test/rules/src/https_everywhere_checker/check_rules.py test/rules/manual.checker.config "$@"

‎test/run_travis.sh‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
#!/bin/bash
2+
set -x
3+
toplevel="$(git rev-parse --show-toplevel)"
4+
testdir="${toplevel}/test/selenium"
5+
srcdir="${toplevel}/chromium"
6+
linter="${toplevel}/utils/eslint/node_modules/.bin/eslint --ignore-path ${srcdir}/.eslintignore"
7+
8+
9+
function run_lint {
10+
$linter $srcdir
11+
if [ $? != 0 ]; then
12+
echo "Linting errors"
13+
exit 1
14+
fi
15+
}
16+
17+
function run_unittests {
18+
pushd ${srcdir}
19+
npm run cover # run with coverage
20+
if [ $? != 0 ]; then
21+
echo "unittest errors"
22+
exit 1
23+
fi
24+
npm run report
25+
popd
26+
}
27+
28+
function run_selenium {
29+
ENABLE_XVFB=1 py.test -v --capture=no ${testdir} # autodiscover and run the tests
30+
}
31+
32+
if [ "$TEST" == "lint" ]; then
33+
echo "running lint tests"
34+
run_lint
35+
elif [ "$TEST" == "unittests" ]; then
36+
echo "Running unittests"
37+
run_unittests
38+
elif [ "$TEST" == "validations" ] || [ "$TEST" == "fetch" ] || [ "$TEST" == "preloaded" ]; then
39+
40+
# Folder paths, relative to parent
41+
RULESETFOLDER="${toplevel}/src/chrome/content/rules"
42+
43+
# Go to git repo root; taken from ../test.sh. Note that
44+
# $GIT_DIR is .git in this case.
45+
if [ -n "$GIT_DIR" ]
46+
then
47+
# $GIT_DIR is set, so we're running as a hook.
48+
cd $GIT_DIR
49+
cd ..
50+
else
51+
# Let's CD to the right place.
52+
cd $toplevel
53+
fi
54+
55+
# Fetch the current GitHub version of HTTPS-E to compare to its master
56+
git remote add upstream-for-travis https://github.com/EFForg/https-everywhere.git
57+
trap 'git remote remove upstream-for-travis' EXIT
58+
59+
# Only do a shallow fetch if we're in Travis. No need otherwise.
60+
if [ $TRAVIS ]; then
61+
git fetch --depth=50 upstream-for-travis master
62+
else
63+
git fetch upstream-for-travis master
64+
fi
65+
66+
COMMON_BASE_COMMIT=$(git merge-base upstream-for-travis/master HEAD)
67+
RULESETS_CHANGED=$(git diff --name-only $COMMON_BASE_COMMIT | grep $RULESETFOLDER | grep '.xml')
68+
if [ "$(git diff --name-only $COMMON_BASE_COMMIT)" != "$RULESETS_CHANGED" ]; then
69+
ONLY_RULESETS_CHANGED=false
70+
fi
71+
72+
# At this point, if anything fails, the test should fail
73+
set -e
74+
75+
if [ "$TEST" == "validations" ]; then
76+
echo >&2 "Performing validations on rulesets."
77+
docker run --rm -ti -v $(pwd):/opt httpse bash -c "test/validations.sh"
78+
fi
79+
80+
if [ "$TEST" == "fetch" ]; then
81+
echo >&2 "Testing test URLs in all changed rulesets."
82+
# --privileged is required here for miredo to create a network tunnel
83+
docker run --rm -ti -v $(pwd):/opt -e RULESETS_CHANGED="$RULESETS_CHANGED" --privileged httpse bash -c "service miredo start && service tor start && test/fetch.sh"
84+
fi
85+
86+
if [ "$TEST" == "preloaded" ]; then
87+
echo >&2 "Ensuring rulesets do not introduce targets which are already HSTS preloaded."
88+
docker run --rm -ti -v $(pwd):/opt -e RULESETS_CHANGED="$RULESETS_CHANGED" node bash -c "cd /opt/utils/hsts-prune && npm install && node index.js"
89+
[ `git diff --name-only $RULESETFOLDER | wc -l` -eq 0 ]
90+
fi
91+
92+
exit 0
93+
94+
else
95+
case $BROWSER in
96+
*chrome*)
97+
echo "running tests on chrome"
98+
run_selenium
99+
;;
100+
*firefox*)
101+
echo "running tests on firefox"
102+
run_selenium
103+
;;
104+
*)
105+
echo "bad TEST variable, got $TEST"
106+
exit 1
107+
;;
108+
esac
109+
fi

setup_travis.sh renamed to test/setup_travis.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ function setup_docker {
4646
docker build -t httpse .
4747
}
4848

49-
case $INFO in
49+
case $TEST in
5050
*chrome*)
5151
setup_chrome
5252
browser_setup
@@ -61,13 +61,13 @@ case $INFO in
6161
*unittests*)
6262
setup_unittests
6363
;;
64-
*rules*|*fetch*)
64+
*validations*|*fetch*)
6565
setup_docker
6666
;;
6767
*preloaded*)
6868
;;
6969
*)
70-
echo "bad INFO variable, got $INFO"
70+
echo "bad TEST variable, got $TEST"
7171
exit 1
7272
;;
7373
esac

0 commit comments

Comments
 (0)