Skip to content

bisect.sh: make it possible to detect commits which fix scan time regressions / improved documentation #4863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danmar merged 1 commit into from
Mar 8, 2023
Merged

bisect.sh: make it possible to detect commits which fix scan time regressions / improved documentation #4863

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions tools/bisect/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,26 @@ Provide the expected error ID (`unreadVariable`) as the `expected` parameter.

## Bisecting scan time regressions

We use daca@home to track differences in scan time. An overview of regressions in scan time can be found at http://cppcheck1.osuosl.org:8000/time_gt.html.
It is also possible to bisect for a regression in scan time.

You need to download the archive as specified by the second line in the output and extract it.
This is done by determinaing the time it took for the "good" commit to finish and setting a timeout twice that size for the calls to determine the "bad" commit.

To bisect these kinds of regressions you currently need to adjust the `bisect.sh` script and set the `hang` variable to appropriate value:<br/>
`1` - find the commit which started the hang<br/>
`2` - find the commit which resolved the hang<br/>

### General notes

As we are currently using the process exitcode to pass the elapsed time to the script it will not work properly with vey long runtime (>= 255 seconds) as it will overflow.

In case the run-time before the regression was very short (<= 1 second) you might need to adjust the `elapsed_time` variable in `bisect.sh` to a higher value to avoid potential false positives.
This might also be necessary to determine one of multiple regressions in the commit range.

After the bisect finished you should take a look at the output and make sure the elpased time of the repective commit looks as expected.

### daca@home notes

We use daca@home to track differences in scan time. An overview of regressions in scan time can be found at http://cppcheck1.osuosl.org:8000/time_gt.html.

If the overall scan time regressed you need to specify the whole folder.

Expand All @@ -97,6 +114,8 @@ If a timeout (potential hang) was introduced you can simply specify the file fro

### Bisecting daca@home issues

You need to download the archive as specified by the second line in the output and extract it.

Use the following data as respective parameters:

`hash-good` the latest tagged release - the second value from the `cppcheck:` line<br/>
Expand Down
13 changes: 8 additions & 5 deletions tools/bisect/bisect.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ expected=$4

# TODO: verify "good" commit happened before "bad" commit

# 0 - regular result based bisect
# 1 - find commit which started hang
# 2 - find commit which resolved hang
hang=0

script_dir="$(dirname "$(realpath "$0")")"
Expand Down Expand Up @@ -48,7 +51,7 @@ git bisect start -- Makefile 'addons/*.py' 'cfg/*.cfg' 'cli/*.cpp' 'cli/*.h' 'ex

git checkout "$hash_good" || exit 1

if [ $hang -eq 1 ]; then
if [ $hang -ne 0 ]; then
# TODO: exitcode overflow on 255
# get expected time from good commit
python3 "$script_dir/bisect_hang.py" "$bisect_dir" "$options"
Expand All @@ -69,8 +72,8 @@ git bisect good || exit 1
git checkout "$hash_bad" || exit 1

# verify the given commit is actually "bad"
if [ $hang -eq 1 ]; then
python3 "$script_dir/bisect_hang.py" "$bisect_dir" "$options" $elapsed_time
if [ $hang -ne 0 ]; then
python3 "$script_dir/bisect_hang.py" "$bisect_dir" "$options" $elapsed_time $hang
else
python3 "$script_dir/bisect_res.py" "$bisect_dir" "$options" "$expected"
fi
Expand All @@ -84,8 +87,8 @@ fi
git bisect bad || exit 1

# perform the actual bisect
if [ $hang -eq 1 ]; then
git bisect run python3 "$script_dir/bisect_hang.py" "$bisect_dir" "$options" $elapsed_time || exit 1
if [ $hang -ne 0 ]; then
git bisect run python3 "$script_dir/bisect_hang.py" "$bisect_dir" "$options" $elapsed_time $hang || exit 1
else
git bisect run python3 "$script_dir/bisect_res.py" "$bisect_dir" "$options" "$expected" || exit 1
fi
Expand Down
10 changes: 7 additions & 3 deletions tools/bisect/bisect_hang.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,14 @@ def run(cppcheck_path, options, elapsed_time=None):
options = sys.argv[2]
if '--error-exitcode=0' not in options:
options += ' --error-exitcode=0'
if len(sys.argv) == 4:
if len(sys.argv) >= 4:
elapsed_time = float(sys.argv[3])
else:
elapsed_time = None
if len(sys.argv) == 5:
invert = sys.argv[4] == "2"
else:
invert = False

try:
cppcheck_path = build_cppcheck(bisect_path)
Expand Down Expand Up @@ -68,8 +72,8 @@ def run(cppcheck_path, options, elapsed_time=None):
sys.exit(EC_SKIP) # error occured

if not run_res:
sys.exit(EC_BAD) # timeout occured
sys.exit(EC_BAD if not invert else EC_GOOD) # timeout occured

print('run_time: {}'.format(run_time))

sys.exit(EC_GOOD) # no timeout
sys.exit(EC_GOOD if not invert else EC_BAD) # no timeout