Skip to content

Commit 6b5abc1

Browse files
gselzergselzer
authored
Add Ops Engine benchmarking module (#104)
* Initial Benchmarking module * Add ImageJ Ops to the mix * Remove module-info.java * BIG WIP: Add benchmarking RTD page * Re-modularize benchmarks Helps for javadoc generation, I think * Make chart-gen generate a copyable output * Flesh out Benchmarking document
1 parent 9490f28 commit 6b5abc1

14 files changed

Lines changed: 1237 additions & 0 deletions

File tree

‎docs/ops/doc/Benchmarks.md‎

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
# SciJava Ops Benchmarks
2+
3+
This page describes a quantitative analysis of the SciJava Ops framework, and is heavily inspired by a similar comparison of [ImgLib2](https://imagej.net/libs/imglib2/benchmarks).
4+
5+
For this analysis, we compare SciJava Ops against:
6+
1) Raw code execution
7+
2) [ImageJ Ops](https://imagej.net/libs/imagej-ops/index)
8+
9+
Some of the charts shown plot execution time as a function of the number of executions - this allows us to show the impact of both the just-in-time (JIT) compiler and the framework's ability to improve performance on repeated function calls.
10+
11+
Other charts plot execution time as a function of input size, which allows analysis of framework overhead.
12+
13+
## Hardware and Software
14+
15+
This analysis was performed with the following hardware:
16+
* 2017 Dell Inspiron 15 7000 Gaming
17+
* Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
18+
* 16 GB 2400 MHz DDR4 RAM
19+
The following software components were used:
20+
* Ubuntu 22.04.3 LTS
21+
* OpenJDK Runtime Environment (build 11.0.21) with OpenJDK 64-Bit Server VM (build 11.0.21, mixed mode, sharing)
22+
* SciJava Ops Engine version `0.0-SNAPSHOT`
23+
* ImageJ Ops version `2.0.0`
24+
25+
## "Cheap" Operations
26+
27+
Following the precedent of ImgLib2, we first analyze the performance of each execution method in performing a simple byte inversion, which can be performed very quickly. This operation lends itself to the analysis of framework overhead.
28+
* The `Raw` execution method simply calls a static method
29+
* The `SciJava Ops` execution method discovers the static method through the `@implNote` Op declaration and calls the functionality through an `OpEnvironment`
30+
* The `ImageJ Ops` execution method discovers the static method through an `@Plugin` wrapper class and calls the functionality through an `OpService`
31+
32+
```java
33+
/**
34+
* @param data the data to invert
35+
* @implNote op name="invert",type=Inplace1
36+
*/
37+
public static void invertRaw(final byte[] data) {
38+
for (int i = 0; i < data.length; i++) {
39+
final int value = data[i] & 0xff;
40+
final int result = 255 - value;
41+
data[i] = (byte) result;
42+
}
43+
}
44+
```
45+
46+
### Repetition Results
47+
48+
We first note that the JIT improves performance benefits starting from the second iteration on all tested methods.
49+
50+
For the very first execution, both SciJava Ops and ImageJ Ops incur decreased performance due to the overhead of discovering and matching the required functionality. The overhead for SciJava Ops is approximately half as much as the overhead of ImageJ Ops.
51+
52+
Starting from the second execution, SciJava Ops rivals the performance of Raw execution, as it caches the results of prior matching calls. ImageJ Ops does not provide similar functionality.
53+
54+
<div>
55+
<div class="dygraph" id="cheapIterationVsTime1" style="width: 50%; float:left"></div>
56+
<div class="dygraph" id="cheapIterationVsTime25" style="width: 50%"></div>
57+
</div>
58+
59+
### Scaling Results
60+
61+
For the first iteration, we see a consistent hierarchy, where Raw execution outperforms SciJava Ops, and SciJava Ops outperforms ImageJ Ops. This hierarchy stems from matching overhead.
62+
63+
For the final iteration, we see that SciJava Ops consistently performs equivalently to Raw execution, while ImageJ Ops maintains worse performance from matching overhead.
64+
65+
<div>
66+
<div class="dygraph" id="cheapResolutionVsTime1" style="width: 50%; float:left"></div>
67+
<div class="dygraph" id="cheapResolutionVsTime10" style="width: 50%"></div>
68+
</div>
69+
70+
## "Expensive" Operations
71+
72+
We now analyze the performance of each execution method in performing randomization. This operation is much more intensive, and allows us to ensure that the computation dominates any framework overhead.
73+
* The `Raw` execution method simply calls a static method
74+
* The `SciJava Ops` execution method discovers the static method through the `@implNote` Op declaration and calls the functionality through an `OpEnvironment`
75+
* The `ImageJ Ops` execution method discovers the static method through an `@Plugin` wrapper class and calls the functionality through an `OpService`
76+
77+
```java
78+
/**
79+
* @param data the data to invert
80+
* @implNote op name="randomize",type=Inplace1
81+
*/
82+
public static void randomizeRaw(final byte[] data) {
83+
for (int i = 0; i < data.length; i++) {
84+
final int value = data[i] & 0xff;
85+
final double result = expensiveOperation(value);
86+
data[i] = (byte) result;
87+
}
88+
}
89+
```
90+
91+
### Repetition Results
92+
93+
For this more expensive operation, we see that computation does indeed dominate any framework overhead for both SciJava Ops and ImageJ Ops. At the smallest input size, we see that ImageJ Ops still shows noticeable matching overhead at every iteration, and SciJava Ops shows noticeable overhead for only the first few iterations.
94+
95+
<div>
96+
<div class="dygraph" id="expensiveIterationVsTime1" style="width: 50%; float:left"></div>
97+
<div class="dygraph" id="expensiveIterationVsTime25" style="width: 50%"></div>
98+
</div>
99+
100+
### Scaling Results
101+
102+
As computation dominates overhead, we see no noticeable difference between the three execution methods.
103+
104+
<div>
105+
<div class="dygraph" id="expensiveResolutionVsTime1" style="width: 50%; float:left"></div>
106+
<div class="dygraph" id="expensiveResolutionVsTime10" style="width: 50%"></div>
107+
</div>
108+
109+
## Reproducing these results
110+
111+
To reproduce these results, take the following steps:
112+
1) Assuming you have a supported Python 3.x installed, you can run `src/main/scripts.benchmark.sh` within the SciJava Ops Benchmarks module. This will generate a file `copyme.txt` in your current directory
113+
2) Copy the contents of `copyme.txt` into the HTML at the bottom of this page, starting at the tag `<!--Begin Pasting Benchmark Data-->`
114+
3) Rebuild the document and view the graphs in a browser
115+
116+
<!-- Populate graphs -->
117+
118+
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/dygraphs@2.1.0/dist/dygraph.min.js"></script>
119+
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/dygraphs@2.1.0/dist/dygraph.min.css" />
120+
<style type="text/css">
121+
.dygraph {
122+
display: inline-block;
123+
max-width: 100%;
124+
width: 435px;
125+
height: 250px;
126+
}
127+
.dygraph-legend {
128+
background-color: rgba(200, 200, 255, 0.75) !important;
129+
padding: 4px;
130+
border: 1px solid #000;
131+
border-radius: 10px;
132+
box-shadow: 4px 4px 4px #888;
133+
pointer-events: none;
134+
width: 12em;
135+
}
136+
.dygraph-legend > span.highlight { background-color: rgba(255, 255, 200, 0.75) !important; }
137+
.dygraph-legend > span.highlight { display: inline; }
138+
</style>
139+
140+
141+
<script type="text/javascript">
142+
function plot(id, title, xlabel, data) {
143+
new Dygraph(document.getElementById(id), data, {
144+
title: title,
145+
titleHeight: 24,
146+
xlabel: xlabel,
147+
ylabel: "Time",
148+
includeZero: true,
149+
labelsSeparateLines: true,
150+
drawPoints: true,
151+
pointSize: 3,
152+
highlightCircleSize: 2,
153+
strokeWidth: 1,
154+
strokeBorderWidth: 1,
155+
highlightSeriesOpts: {
156+
strokeWidth: 3,
157+
strokeBorderWidth: 1,
158+
highlightCircleSize: 5
159+
}
160+
});
161+
}
162+
163+
<!--Begin Pasting Benchmark Data-->
164+
plot("cheapIterationVsTime1", "Iteration x Time (ms) at 1 Mpx", "Iteration",
165+
"Iteration,ImageJ Ops,Raw,SciJava Ops\n" +
166+
"0,119,11,79\n" +
167+
"1,15,1,0\n" +
168+
"2,11,0,0\n" +
169+
"3,10,1,0\n" +
170+
"4,10,0,1\n" +
171+
"5,13,1,0\n" +
172+
"6,10,0,2\n" +
173+
"7,7,0,1\n" +
174+
"8,7,0,0\n" +
175+
"9,7,0,0");
176+
177+
plot("cheapIterationVsTime25", "Iteration x Time (ms) at 25 Mpx", "Iteration",
178+
"Iteration,ImageJ Ops,Raw,SciJava Ops\n" +
179+
"0,129,76,84\n" +
180+
"1,21,4,4\n" +
181+
"2,22,4,4\n" +
182+
"3,25,7,6\n" +
183+
"4,12,9,4\n" +
184+
"5,10,4,4\n" +
185+
"6,14,4,3\n" +
186+
"7,12,4,6\n" +
187+
"8,20,5,6\n" +
188+
"9,13,5,6");
189+
190+
plot("cheapResolutionVsTime1", "Resolution x Time (ms) at iteration #1", "Mpx",
191+
"Mpx,ImageJ Ops,Raw,SciJava Ops\n" +
192+
"0,119,11,79\n" +
193+
"1,118,20,67\n" +
194+
"2,122,34,100\n" +
195+
"3,122,14,67\n" +
196+
"4,118,60,58\n" +
197+
"5,118,26,85\n" +
198+
"6,116,100,59\n" +
199+
"7,113,23,66\n" +
200+
"8,129,76,84");
201+
202+
plot("cheapResolutionVsTime10", "Resolution x Time (ms) at iteration #10", "Mpx",
203+
"Mpx,ImageJ Ops,Raw,SciJava Ops\n" +
204+
"0,7,0,0\n" +
205+
"1,7,1,1\n" +
206+
"2,11,1,2\n" +
207+
"3,8,3,3\n" +
208+
"4,11,3,2\n" +
209+
"5,10,3,3\n" +
210+
"6,8,3,4\n" +
211+
"7,11,5,7\n" +
212+
"8,13,5,6");
213+
214+
plot("expensiveIterationVsTime1", "Iteration x Time (ms) at 1 Mpx", "Iteration",
215+
"Iteration,ImageJ Ops,Raw,SciJava Ops\n" +
216+
"0,61,60,50\n" +
217+
"1,40,41,33\n" +
218+
"2,39,33,34\n" +
219+
"3,37,33,32\n" +
220+
"4,34,30,29\n" +
221+
"5,33,29,27\n" +
222+
"6,30,26,26\n" +
223+
"7,29,25,23\n" +
224+
"8,27,23,23\n" +
225+
"9,27,23,22");
226+
227+
plot("expensiveIterationVsTime25", "Iteration x Time (ms) at 25 Mpx", "Iteration",
228+
"Iteration,ImageJ Ops,Raw,SciJava Ops\n" +
229+
"0,798,837,783\n" +
230+
"1,790,784,781\n" +
231+
"2,785,780,774\n" +
232+
"3,761,756,757\n" +
233+
"4,731,730,731\n" +
234+
"5,699,694,693\n" +
235+
"6,649,645,640\n" +
236+
"7,599,590,594\n" +
237+
"8,570,566,566\n" +
238+
"9,557,551,550");
239+
240+
plot("expensiveResolutionVsTime1", "Resolution x Time (ms) at iteration #1", "Mpx",
241+
"Mpx,ImageJ Ops,Raw,SciJava Ops\n" +
242+
"0,61,60,50\n" +
243+
"1,139,205,124\n" +
244+
"2,235,304,221\n" +
245+
"3,331,368,307\n" +
246+
"4,424,466,407\n" +
247+
"5,513,557,488\n" +
248+
"6,596,601,592\n" +
249+
"7,700,743,674\n" +
250+
"8,798,837,783");
251+
252+
plot("expensiveResolutionVsTime10", "Resolution x Time (ms) at iteration #10", "Mpx",
253+
"Mpx,ImageJ Ops,Raw,SciJava Ops\n" +
254+
"0,27,23,22\n" +
255+
"1,92,87,87\n" +
256+
"2,160,156,154\n" +
257+
"3,222,217,218\n" +
258+
"4,293,287,286\n" +
259+
"5,358,352,346\n" +
260+
"6,424,420,416\n" +
261+
"7,480,478,478\n" +
262+
"8,557,551,550");
263+
<!--End Pasting Benchmark Data-->
264+
</script>

‎docs/ops/doc/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ The combination of these libraries allows declarative image analysis workflows,
2222
CallingOps
2323
SearchingForOps
2424
ScriptingInImageJ2
25+
Benchmarks
2526

2627
.. toctree::
2728
:maxdepth: 2

‎pom.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@
5858
<module>scijava/scijava-function</module>
5959
<module>scijava/scijava-meta</module>
6060
<module>scijava/scijava-ops-api</module>
61+
<module>scijava/scijava-ops-benchmarks</module>
6162
<module>scijava/scijava-ops-engine</module>
6263
<module>scijava/scijava-ops-indexer</module>
6364
<module>scijava/scijava-ops-spi</module>

‎scijava/scijava-ops-benchmarks/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
/.apt_generated/
2+
/.apt_generated_tests/

‎scijava/scijava-ops-benchmarks/LICENSE.txt‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
Copyright (c) 2021 - 2023, SciJava developers.
2+
All rights reserved.
3+
4+
Redistribution and use in source and binary forms, with or without modification,
5+
are permitted provided that the following conditions are met:
6+
7+
1. Redistributions of source code must retain the above copyright notice, this
8+
list of conditions and the following disclaimer.
9+
10+
2. Redistributions in binary form must reproduce the above copyright notice,
11+
this list of conditions and the following disclaimer in the documentation
12+
and/or other materials provided with the distribution.
13+
14+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
15+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
16+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
17+
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
18+
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
19+
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
20+
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
21+
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
22+
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23+
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
24+
POSSIBILITY OF SUCH DAMAGE.

‎scijava/scijava-ops-benchmarks/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# SciJava Ops Benchmarks: benchmarking utilities for the SciJava Ops Engine library
2+
3+
TODO

0 commit comments

Comments
 (0)