Skip to content

Create thread dumps for all running JVMs when the build gets cancelled#13100

Merged
normanmaurer merged 3 commits intonetty:4.1from
lhotari:lh-thread-dumps-on-cancel
Jan 9, 2023
Merged

Create thread dumps for all running JVMs when the build gets cancelled#13100
normanmaurer merged 3 commits intonetty:4.1from
lhotari:lh-thread-dumps-on-cancel

Conversation

@lhotari
Copy link
Copy Markdown
Contributor

@lhotari lhotari commented Jan 5, 2023

Motivation:

When a GitHub Actions build job gets cancelled because of a timeout, it would be useful to have a thread dump for all running JVM processes so that it would be able to find the reason why the build job timed out.

Modification:

Add an inline composite action that uses bash, jps and jcmd to create thread dumps for all running Java processes.
This is referenced in ci-pr.yml workflow's verify-pr job and gets run when the build job is cancelled.
This is an example of a thread dump that gets created: https://github.com/lhotari/netty/pull/1/checks#step:7:28 .

Copy link
Copy Markdown
Member

@chrisvest chrisvest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool

Comment thread .github/actions/thread-dump-jvms/action.yml Outdated
@normanmaurer
Copy link
Copy Markdown
Member

@lhotari nice... do you also have a solution for windows / macOs ?

@lhotari
Copy link
Copy Markdown
Contributor Author

lhotari commented Jan 6, 2023

@lhotari nice... do you also have a solution for windows / macOs ?

@normanmaurer I'll work on that.

I also noticed that quite a few builds run the JVM within docker. I'll work on a solution that uses https://github.com/apangin/jattach for triggering the thread dumps since that tool works also for Java processes that run inside docker containers.

@lhotari lhotari requested a review from chrisvest January 9, 2023 09:02
@lhotari
Copy link
Copy Markdown
Contributor Author

lhotari commented Jan 9, 2023

@normanmaurer I have made changes to support java processes in docker containers as well as support for windows / macos.

Here's an example of the thread dump of a process running in a docker container: https://github.com/lhotari/netty/actions/runs/3872145028/jobs/6600659632#step:9:432

This example is from a windows container:
https://github.com/lhotari/netty/actions/runs/3872145028/jobs/6600659012#step:6:42

A small detail on windows is that the maven process gets killed when the GitHub Actions job gets cancelled and there will be no thread dump for the main maven process itself. On Linux, the maven main process isn't killed immediately and there will also be a thread dump available for that process. The thread dump for the surefire process will be available also on windows as seen in the above example.

@normanmaurer
Copy link
Copy Markdown
Member

@lhotari thanks so much! This is amazing

@normanmaurer normanmaurer added this to the 4.1.87.Final milestone Jan 9, 2023
@normanmaurer normanmaurer merged commit e19860a into netty:4.1 Jan 9, 2023
normanmaurer pushed a commit that referenced this pull request Jan 9, 2023
#13100)

Motivation:

When a GitHub Actions build job gets cancelled because of a timeout, it would be useful to have a thread dump for all running JVM processes so that it would be able to find the reason why the build job timed out.

Modification:

Add an inline composite action that uses bash, jps and jcmd to create thread dumps for all running Java processes. 
This is referenced in ci-pr.yml workflow's verify-pr job and gets run when the build job is cancelled.
This is an example of a thread dump that gets created: https://github.com/lhotari/netty/pull/1/checks#step:7:28 .

Result:

Easier to debug builds that did timeout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants