Skip to content

Commit c7148c3

Browse files
committed
Heap Dump to Volume Service
This change adds support for configuring the jvmkill agent to create terminal heap dumps into volume services with a name, label, or tag containing heap- dump. If that service does not exist, the agent continues to print the heap histogram to the console. [resolves cloudfoundry#439]
1 parent 7077581 commit c7148c3

8 files changed

Lines changed: 293 additions & 25 deletions

File tree

docs/jre-open_jdk_jre.md

Lines changed: 50 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,15 @@ The OpenJDK JRE provides Java runtimes from the [OpenJDK][] project. Versions o
44
<table>
55
<tr>
66
<td><strong>Detection Criterion</strong></td>
7-
<td>Unconditional</td>
7+
<td>Unconditional. Existence of a single bound Volume Service will result in Terminal heap dumps being written.
8+
<ul>
9+
<li>Existence of a Volume Service service is defined as the <a href="http://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES"><code>VCAP_SERVICES</code></a> payload containing a service who's name, label or tag has <code>heap-dump</code> as a substring.</li>
10+
</ul>
11+
</td>
812
</tr>
913
<tr>
1014
<td><strong>Tags</strong></td>
11-
<td><tt>open-jdk=&lang;version&rang;, open-jdk-like-memory-calculator=&lang;version&rang;</tt></td>
15+
<td><tt>open-jdk=&lang;version&rang;, open-jdk-like-memory-calculator=&lang;version&rang;, jvmkill=&lang;version&rang;</tt></td>
1216
</tr>
1317
</table>
1418
Tags are printed to standard output by the buildpack detect script
@@ -20,9 +24,11 @@ The JRE can be configured by modifying the [`config/open_jdk_jre.yml`][] file in
2024

2125
| Name | Description
2226
| ---- | -----------
27+
| `jre.repository_root` | The URL of the OpenJDK repository index ([details][repositories]).
28+
| `jre.version` | The version of Java runtime to use. Candidate versions can be found in the listings for [mountainlion][] and [trusty][]. Note: version 1.8.0 and higher require the `memory_sizes` and `memory_heuristics` mappings to specify `metaspace` rather than `permgen`.
29+
| `jvmkill.repository_root` | The URL of the `jvmkill` repository index ([details][repositories]).
30+
| `jvmkill.version` | The version of `jvmkill` to use. Candidate versions can be found in the listings for [mountainlion][jvmkill-mountainlion] and [trusty][jvmkill-trusty].
2331
| `memory_calculator` | Memory calculator defaults, described below under "Memory".
24-
| `repository_root` | The URL of the OpenJDK repository index ([details][repositories]).
25-
| `version` | The version of Java runtime to use. Candidate versions can be found in the listings for [mountainlion][], [precise][], and [trusty][]. Note: version 1.8.0 and higher require the `memory_sizes` and `memory_heuristics` mappings to specify `metaspace` rather than `permgen`.
2632

2733
### Additional Resources
2834
The JRE can also be configured by overlaying a set of resources on the default distribution. To do this, add files to the `resources/open_jdk_jre` directory in the buildpack fork.
@@ -33,6 +39,42 @@ To add the JCE Unlimited Strength `local_policy.jar`, add your file to `resource
3339
#### Custom CA Certificates
3440
To add custom SSL certificates, add your `cacerts` file to `resources/open_jdk_jre/lib/security/cacerts`. This file will be overlayed onto the OpenJDK distribution.
3541

42+
### `jvmkill`
43+
The `jvmkill` agent runs when an application has experience a resource exhaustion event. When this event occurs, the agent will print out a histogram of the first 100 largest types by total number of bytes.
44+
45+
```plain
46+
Resource exhaustion event: the JVM was unable to allocate memory from the heap.
47+
ResourceExhausted! (1/0)
48+
| Instance Count | Total Bytes | Class Name |
49+
| 18273 | 313157136 | [B |
50+
| 47806 | 7648568 | [C |
51+
| 14635 | 1287880 | Ljava/lang/reflect/Method; |
52+
| 46590 | 1118160 | Ljava/lang/String; |
53+
| 8413 | 938504 | Ljava/lang/Class; |
54+
| 28573 | 914336 | Ljava/util/concurrent/ConcurrentHashMap$Node; |
55+
```
56+
57+
It will also print out a summary of all of the memory spaces in the JVM.
58+
59+
```plain
60+
Memory usage:
61+
Heap memory: init 65011712, used 332392888, committed 351797248, max 351797248
62+
Non-heap memory: init 2555904, used 63098592, committed 64815104, max 377790464
63+
Memory pool usage:
64+
Code Cache: init 2555904, used 14702208, committed 15007744, max 251658240
65+
PS Eden Space: init 16252928, used 84934656, committed 84934656, max 84934656
66+
PS Survivor Space: init 2621440, used 0, committed 19398656, max 19398656
67+
Compressed Class Space: init 0, used 5249512, committed 5505024, max 19214336
68+
Metaspace: init 0, used 43150616, committed 44302336, max 106917888
69+
PS Old Gen: init 43515904, used 247459792, committed 247463936, max 247463936
70+
```
71+
72+
If a heap dump [Volume Service][] is bound, terminal heap dumps will be written with the pattern `"<CONTAINER_DIR>/<SPACE_NAME>-<SPACE_ID>/<APPLICATION_NAME>-<APPLICATION_ID>/<INSTANCE_INDEX>-<TIMESTAMP>-<INSTANCE_ID>.hprof`
73+
74+
```plain
75+
Heapdump written to /var/vcap/data/9ae0b817-1446-4915-9990-74c1bb26f147/pcfdev-space-e91c5c39-b546-41d9-8095-9a45fa65df9e/java-main-application-892f20ab-9a53-441d-be3e-72c38f2a1055/0-2017-06-13T18:31:29+0000-7b23124e-7f0f-4a08-457b-60802d0a7326.hprof
76+
```
77+
3678
### Memory
3779
The total available memory for the application's container is specified when an application is pushed.
3880
The Java buildpack uses this value to control the JRE's use of various
@@ -84,7 +126,7 @@ The container's total available memory is allocated into heap, metaspace and com
84126
direct memory, and stack memory settings.
85127

86128
The memory calculation is described in more detail in the [Memory Calculator's README].
87-
129+
88130
The inputs to the memory calculation, except the container's total memory (which is unknown at staging time), are logged during staging, for example:
89131
```
90132
Loaded Classes: 13974, Threads: 300, JAVA_OPTS: ''
@@ -105,10 +147,12 @@ JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=99199
105147
[`config/open_jdk_jre.yml`]: ../config/open_jdk_jre.yml
106148
[Configuration and Extension]: ../README.md#configuration-and-extension
107149
[Java Buildpack Memory Calculator]: https://github.com/cloudfoundry/java-buildpack-memory-calculator
150+
[jvmkill-mountainlion]: http://download.pivotal.io.s3.amazonaws.com/jvmkill/mountainlion/x86_64/index.yml
151+
[jvmkill-trusty]: http://download.pivotal.io.s3.amazonaws.com/jvmkill/trusty/x86_64/index.yml
108152
[Memory Calculator's README]: https://github.com/cloudfoundry/java-buildpack-memory-calculator
109153
[mountainlion]: http://download.pivotal.io.s3.amazonaws.com/openjdk/mountainlion/x86_64/index.yml
110154
[OpenJDK]: http://openjdk.java.net
111-
[precise]: http://download.pivotal.io.s3.amazonaws.com/openjdk/precise/x86_64/index.yml
112155
[repositories]: extending-repositories.md
113156
[trusty]: http://download.pivotal.io.s3.amazonaws.com/openjdk/trusty/x86_64/index.yml
114157
[version syntax]: extending-repositories.md#version-syntax-and-ordering
158+
[Volume Service]: https://docs.cloudfoundry.org/devguide/services/using-vol-services.html

docs/jre-oracle_jre.md

Lines changed: 50 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,15 @@ The Oracle JRE provides Java runtimes from [Oracle][] project. No versions of t
44
<table>
55
<tr>
66
<td><strong>Detection Criterion</strong></td>
7-
<td>Unconditional</td>
7+
<td>Unconditional. Existence of a single bound Volume Service will result in Terminal heap dumps being written.
8+
<ul>
9+
<li>Existence of a Volume Service service is defined as the <a href="http://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES"><code>VCAP_SERVICES</code></a> payload containing a service who's name, label or tag has <code>heap-dump</code> as a substring.</li>
10+
</ul>
11+
</td>
812
</tr>
913
<tr>
1014
<td><strong>Tags</strong></td>
11-
<td><tt>oracle=&lang;version&rang;, open-jdk-like-memory-calculator=&lang;version&rang;</tt></td>
15+
<td><tt>oracle=&lang;version&rang;, open-jdk-like-memory-calculator=&lang;version&rang;, jvmkill=&lang;version&rang;</tt></td>
1216
</tr>
1317
</table>
1418
Tags are printed to standard output by the buildpack detect script
@@ -36,9 +40,11 @@ To use Oracle JRE instead of OpenJDK without forking java-buildpack, set environ
3640

3741
| Name | Description
3842
| ---- | -----------
43+
| `jre.repository_root` | The URL of the Oracle repository index ([details][repositories]).
44+
| `jre.version` | The version of Java runtime to use. Candidate versions can be found in the the repository that you have created to house the JREs. Note: version 1.8.0 and higher require the `memory_sizes` and `memory_heuristics` mappings to specify `metaspace` rather than `permgen`.
45+
| `jvmkill.repository_root` | The URL of the `jvmkill` repository index ([details][repositories]).
46+
| `jvmkill.version` | The version of `jvmkill` to use. Candidate versions can be found in the listings for [mountainlion][jvmkill-mountainlion] and [trusty][jvmkill-trusty].
3947
| `memory_calculator` | Memory calculator defaults, described below under "Memory".
40-
| `repository_root` | The URL of the Oracle repository index ([details][repositories]).
41-
| `version` | The version of Java runtime to use. Candidate versions can be found in the the repository that you have created to house the JREs. Note: version 1.8.0 and higher require the `memory_sizes` and `memory_heuristics` mappings to specify `metaspace` rather than `permgen`.
4248

4349
### Additional Resources
4450
The JRE can also be configured by overlaying a set of resources on the default distribution. To do this, add files to the `resources/oracle_jre` directory in the buildpack fork.
@@ -49,6 +55,42 @@ To add the JCE Unlimited Strength `local_policy.jar`, add your file to `resource
4955
#### Custom CA Certificates
5056
To add custom SSL certificates, add your `cacerts` file to `resources/oracle_jre/lib/security/cacerts`. This file will be overlayed onto the Oracle distribution.
5157

58+
### `jvmkill`
59+
The `jvmkill` agent runs when an application has experience a resource exhaustion event. When this event occurs, the agent will print out a histogram of the first 100 largest types by total number of bytes.
60+
61+
```plain
62+
Resource exhaustion event: the JVM was unable to allocate memory from the heap.
63+
ResourceExhausted! (1/0)
64+
| Instance Count | Total Bytes | Class Name |
65+
| 18273 | 313157136 | [B |
66+
| 47806 | 7648568 | [C |
67+
| 14635 | 1287880 | Ljava/lang/reflect/Method; |
68+
| 46590 | 1118160 | Ljava/lang/String; |
69+
| 8413 | 938504 | Ljava/lang/Class; |
70+
| 28573 | 914336 | Ljava/util/concurrent/ConcurrentHashMap$Node; |
71+
```
72+
73+
It will also print out a summary of all of the memory spaces in the JVM.
74+
75+
```plain
76+
Memory usage:
77+
Heap memory: init 65011712, used 332392888, committed 351797248, max 351797248
78+
Non-heap memory: init 2555904, used 63098592, committed 64815104, max 377790464
79+
Memory pool usage:
80+
Code Cache: init 2555904, used 14702208, committed 15007744, max 251658240
81+
PS Eden Space: init 16252928, used 84934656, committed 84934656, max 84934656
82+
PS Survivor Space: init 2621440, used 0, committed 19398656, max 19398656
83+
Compressed Class Space: init 0, used 5249512, committed 5505024, max 19214336
84+
Metaspace: init 0, used 43150616, committed 44302336, max 106917888
85+
PS Old Gen: init 43515904, used 247459792, committed 247463936, max 247463936
86+
```
87+
88+
If a heap dump [Volume Service][] is bound, terminal heap dumps will be written with the pattern `"<CONTAINER_DIR>/<SPACE_NAME>-<SPACE_ID>/<APPLICATION_NAME>-<APPLICATION_ID>/<INSTANCE_INDEX>-<TIMESTAMP>-<INSTANCE_ID>.hprof`
89+
90+
```plain
91+
Heapdump written to /var/vcap/data/9ae0b817-1446-4915-9990-74c1bb26f147/pcfdev-space-e91c5c39-b546-41d9-8095-9a45fa65df9e/java-main-application-892f20ab-9a53-441d-be3e-72c38f2a1055/0-2017-06-13T18:31:29+0000-7b23124e-7f0f-4a08-457b-60802d0a7326.hprof
92+
```
93+
5294
### Memory
5395
The total available memory for the application's container is specified when an application is pushed.
5496
The Java buildpack uses this value to control the JRE's use of various
@@ -100,7 +142,7 @@ The container's total available memory is allocated into heap, metaspace and com
100142
direct memory, and stack memory settings.
101143

102144
The memory calculation is described in more detail in the [Memory Calculator's README].
103-
145+
104146
The inputs to the memory calculation, except the container's total memory (which is unknown at staging time), are logged during staging, for example:
105147
```
106148
Loaded Classes: 13974, Threads: 300, JAVA_OPTS: ''
@@ -122,8 +164,11 @@ JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=99199
122164
[`config/oracle_jre.yml`]: ../config/oracle_jre.yml
123165
[Configuration and Extension]: ../README.md#configuration-and-extension
124166
[Java Buildpack Memory Calculator]: https://github.com/cloudfoundry/java-buildpack-memory-calculator
167+
[jvmkill-mountainlion]: http://download.pivotal.io.s3.amazonaws.com/jvmkill/mountainlion/x86_64/index.yml
168+
[jvmkill-trusty]: http://download.pivotal.io.s3.amazonaws.com/jvmkill/trusty/x86_64/index.yml
125169
[Memory Calculator's README]: https://github.com/cloudfoundry/java-buildpack-memory-calculator
126170
[OpenJDK JRE]: jre-open_jdk_jre.md
127171
[Oracle]: http://www.oracle.com/technetwork/java/index.html
128172
[repositories]: extending-repositories.md
129173
[version syntax]: extending-repositories.md#version-syntax-and-ordering
174+
[Volume Service]: https://docs.cloudfoundry.org/devguide/services/using-vol-services.html

docs/jre-zulu_jre.md

Lines changed: 52 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,15 @@ Azul Zulu JRE provides Java runtimes developed by Azul team. Versions of Java f
44
<table>
55
<tr>
66
<td><strong>Detection Criterion</strong></td>
7-
<td>Unconditional</td>
7+
<td>Unconditional. Existence of a single bound Volume Service will result in Terminal heap dumps being written.
8+
<ul>
9+
<li>Existence of a Volume Service service is defined as the <a href="http://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES"><code>VCAP_SERVICES</code></a> payload containing a service who's name, label or tag has <code>heap-dump</code> as a substring.</li>
10+
</ul>
11+
</td>
812
</tr>
913
<tr>
1014
<td><strong>Tags</strong></td>
11-
<td><tt>open-jdk-like-jre=&lang;version&rang;, open-jdk-like-memory-calculator=&lang;version&rang;</tt></td>
15+
<td><tt>open-jdk-like-jre=&lang;version&rang;, open-jdk-like-memory-calculator=&lang;version&rang;, jvmkill=&lang;version&rang;</tt></td>
1216
</tr>
1317
</table>
1418
Tags are printed to standard output by the buildpack detect script.
@@ -27,9 +31,11 @@ To use Zulu JRE instead of OpenJDK without forking java-buildpack, set environme
2731

2832
| Name | Description
2933
| ---- | -----------
34+
| `jre.repository_root` | The URL of the Zulu repository index ([details][repositories]).
35+
| `jre.version` | The version of Java runtime to use. Note: version 1.8.0 and higher require the `memory_sizes` and `memory_heuristics` mappings to specify `metaspace` rather than `permgen`.
36+
| `jvmkill.repository_root` | The URL of the `jvmkill` repository index ([details][repositories]).
37+
| `jvmkill.version` | The version of `jvmkill` to use. Candidate versions can be found in the listings for [mountainlion][jvmkill-mountainlion] and [trusty][jvmkill-trusty].
3038
| `memory_calculator` | Memory calculator defaults, described below under "Memory".
31-
| `repository_root` | The URL of the Zulu repository index ([details][repositories]).
32-
| `version` | The version of Java runtime to use. Note: version 1.8.0 and higher require the `memory_sizes` and `memory_heuristics` mappings to specify `metaspace` rather than `permgen`.
3339

3440
### Additional Resources
3541
The JRE can also be configured by overlaying a set of resources on the default distribution. To do this, add files to the `resources/zulu_jre` directory in the buildpack fork.
@@ -40,6 +46,42 @@ To add the JCE Unlimited Strength `local_policy.jar`, add your file to `resource
4046
#### Custom CA Certificates
4147
To add custom SSL certificates, add your `cacerts` file to `resources/zulu_jre/lib/security/cacerts`. This file will be overlayed onto the Zulu distribution.
4248

49+
### `jvmkill`
50+
The `jvmkill` agent runs when an application has experience a resource exhaustion event. When this event occurs, the agent will print out a histogram of the first 100 largest types by total number of bytes.
51+
52+
```plain
53+
Resource exhaustion event: the JVM was unable to allocate memory from the heap.
54+
ResourceExhausted! (1/0)
55+
| Instance Count | Total Bytes | Class Name |
56+
| 18273 | 313157136 | [B |
57+
| 47806 | 7648568 | [C |
58+
| 14635 | 1287880 | Ljava/lang/reflect/Method; |
59+
| 46590 | 1118160 | Ljava/lang/String; |
60+
| 8413 | 938504 | Ljava/lang/Class; |
61+
| 28573 | 914336 | Ljava/util/concurrent/ConcurrentHashMap$Node; |
62+
```
63+
64+
It will also print out a summary of all of the memory spaces in the JVM.
65+
66+
```plain
67+
Memory usage:
68+
Heap memory: init 65011712, used 332392888, committed 351797248, max 351797248
69+
Non-heap memory: init 2555904, used 63098592, committed 64815104, max 377790464
70+
Memory pool usage:
71+
Code Cache: init 2555904, used 14702208, committed 15007744, max 251658240
72+
PS Eden Space: init 16252928, used 84934656, committed 84934656, max 84934656
73+
PS Survivor Space: init 2621440, used 0, committed 19398656, max 19398656
74+
Compressed Class Space: init 0, used 5249512, committed 5505024, max 19214336
75+
Metaspace: init 0, used 43150616, committed 44302336, max 106917888
76+
PS Old Gen: init 43515904, used 247459792, committed 247463936, max 247463936
77+
```
78+
79+
If a heap dump [Volume Service][] is bound, terminal heap dumps will be written with the pattern `"<CONTAINER_DIR>/<SPACE_NAME>-<SPACE_ID>/<APPLICATION_NAME>-<APPLICATION_ID>/<INSTANCE_INDEX>-<TIMESTAMP>-<INSTANCE_ID>.hprof`
80+
81+
```plain
82+
Heapdump written to /var/vcap/data/9ae0b817-1446-4915-9990-74c1bb26f147/pcfdev-space-e91c5c39-b546-41d9-8095-9a45fa65df9e/java-main-application-892f20ab-9a53-441d-be3e-72c38f2a1055/0-2017-06-13T18:31:29+0000-7b23124e-7f0f-4a08-457b-60802d0a7326.hprof
83+
```
84+
4385
### Memory
4486
The total available memory for the application's container is specified when an application is pushed.
4587
The Java buildpack uses this value to control the JRE's use of various
@@ -91,7 +133,7 @@ The container's total available memory is allocated into heap, metaspace and com
91133
direct memory, and stack memory settings.
92134

93135
The memory calculation is described in more detail in the [Memory Calculator's README].
94-
136+
95137
The inputs to the memory calculation, except the container's total memory (which is unknown at staging time), are logged during staging, for example:
96138
```
97139
Loaded Classes: 13974, Threads: 300, JAVA_OPTS: ''
@@ -111,10 +153,13 @@ JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=99199
111153

112154
[`config/components.yml`]: ../config/components.yml
113155
[`config/zulu_jre.yml`]: ../config/zulu_jre.yml
156+
[Azul Zulu]: https://www.azul.com/products/zulu/
114157
[Configuration and Extension]: ../README.md#configuration-and-extension
115158
[Java Buildpack Memory Calculator]: https://github.com/cloudfoundry/java-buildpack-memory-calculator
159+
[jvmkill-mountainlion]: http://download.pivotal.io.s3.amazonaws.com/jvmkill/mountainlion/x86_64/index.yml
160+
[jvmkill-trusty]: http://download.pivotal.io.s3.amazonaws.com/jvmkill/trusty/x86_64/index.yml
116161
[Memory Calculator's README]: https://github.com/cloudfoundry/java-buildpack-memory-calculator
117-
[Zulu JRE]: jre-zulu_jre.md
118-
[Azul Zulu]: https://www.azul.com/products/zulu/
119162
[repositories]: extending-repositories.md
120163
[version syntax]: extending-repositories.md#version-syntax-and-ordering
164+
[Volume Service]: https://docs.cloudfoundry.org/devguide/services/using-vol-services.html
165+
[Zulu JRE]: jre-zulu_jre.md

0 commit comments

Comments
 (0)