Skip to content

chore: skip some flaky tests#1643

Merged
johnstcn merged 3 commits into
mainfrom
cj/test-flake-skip
May 20, 2022
Merged

chore: skip some flaky tests#1643
johnstcn merged 3 commits into
mainfrom
cj/test-flake-skip

Conversation

@johnstcn
Copy link
Copy Markdown
Member

@johnstcn johnstcn commented May 20, 2022

I ran gotestsum a few times and skipped some tests that flaked out.
This is not a long-term solution. We still need to fix these.

@johnstcn johnstcn requested a review from a team May 20, 2022 21:01
AbhineetJain
AbhineetJain approved these changes May 20, 2022
View reviewed changes
Comment thread cli/ssh_test.go
)

func TestSSH(t *testing.T) {
t.Skip("This is causing test flakes. TODO @cian fix this")
Copy link
Copy Markdown
Contributor

@AbhineetJain AbhineetJain May 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an issue filed for this? Maybe link it in a comment?

Comment thread peer/conn_test.go Outdated
johnstcn
johnstcn commented May 20, 2022
View reviewed changes
Comment thread peer/conn_test.go Outdated
@greyscaled
Copy link
Copy Markdown
Contributor

greyscaled commented May 20, 2022

It looks like the test/go/postgres check is flaking out in this PR too 😢 . I'm not sure if the suppression here were for that or other tests. I hope we can use a similar tactic to hunt down the flakes in that check, or at least only run it when the data layer changes.

postgres

@johnstcn
Copy link
Copy Markdown
Member Author

johnstcn commented May 20, 2022

It looks like the test/go/postgres check is flaking out in this PR too 😢

make test is not the same as what gets run in test/go/postgres!

@johnstcn
Copy link
Copy Markdown
Member Author

johnstcn commented May 20, 2022

When I ran the postgres tests locally, they were working fine with a postgresql service running via Homebrew.
After I tried running the postgresql server in Docker, however, it ended up crashing at one point with this error:

2022-05-20 22:08:45.386 UTC [2168] LOG:  unexpected EOF on client connection with an open transaction
2022-05-20 22:08:45.439 UTC [1] LOG:  checkpointer process (PID 70) was terminated by signal 9: Killed
2022-05-20 22:08:45.439 UTC [1] LOG:  terminating any other active server processes
2022-05-20 22:08:45.439 UTC [2172] WARNING:  terminating connection because of crash of another server process
2022-05-20 22:08:45.439 UTC [2172] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-20 22:08:45.439 UTC [2172] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-20 22:08:45.439 UTC [2169] WARNING:  terminating connection because of crash of another server process
2022-05-20 22:08:45.439 UTC [2169] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-20 22:08:45.439 UTC [2169] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-20 22:08:45.440 UTC [73] WARNING:  terminating connection because of crash of another server process
2022-05-20 22:08:45.440 UTC [73] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-20 22:08:45.440 UTC [73] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-20 22:08:45.441 UTC [2173] WARNING:  terminating connection because of crash of another server process
2022-05-20 22:08:45.441 UTC [2173] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-20 22:08:45.441 UTC [2173] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-20 22:08:45.472 UTC [1] LOG:  all server processes terminated; reinitializing

@johnstcn
Copy link
Copy Markdown
Member Author

johnstcn commented May 20, 2022 •
edited
Loading

I'm experimenting reducing the level of parallelism in the postgres tests.
I'm finding that it's surprisingly easy for postgres to get OOM-killed when you run lots of tests at once. I think I saw the number of postgres processes peak at 137 or so (!)

@johnstcn
Copy link
Copy Markdown
Member Author

johnstcn commented May 20, 2022

I also added some Makefile targets so we can more easily run the full-fat test suite and dig into these sorts of things locally.

@johnstcn johnstcn merged commit a8a8f9d into main May 20, 2022
@johnstcn johnstcn deleted the cj/test-flake-skip branch May 20, 2022 23:39
kylecarbs pushed a commit that referenced this pull request Jun 10, 2022
@johnstcn @kylecarbs
chore: skip some flaky tests (#1643)
1ac99d4 
* chore: skip some flaky tests

* Update peer/conn_test.go

* add makefile targets, reduce parallelism in go test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@AbhineetJain AbhineetJain AbhineetJain approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@johnstcn @greyscaled @AbhineetJain