performance-metrics: Add --continue-on-failure flag and status tracking by anirudhrb · Pull Request #7905 · cloud-hypervisor/cloud-hypervisor

anirudhrb · 2026-03-26T10:30:20Z

Add a --continue-on-failure CLI flag that allows the test harness to continue executing remaining tests after encountering a failure, instead of aborting immediately. When set, failed tests are recorded with zeroed metrics and a "FAILED" status, the report file is always generated, and the process exits with a non-zero code if any test failed.

Without the flag, the existing fail-fast behavior is preserved.

Also add a "status" field ("PASSED"/"FAILED") to PerformanceTestResult so report consumers can distinguish successful tests from failed ones.

weltling · 2026-03-26T12:59:18Z

performance-metrics/src/main.rs

+                        max: 0.0,
+                        min: 0.0,
+                        status: "FAILED".to_string(),
+                    });


A panicked test could leave things hanging. The failure path should call cleanup_stale_processes(), to avoid possible subsequent tests contamination.

Thanks

weltling · 2026-03-26T13:04:14Z

performance-metrics/src/main.rs

    std_dev: f64,
    max: f64,
    min: f64,
+    status: String,


If there are no compatibility concerns regarding the JSON format change, at least this should update the ./docs/perfomance-metrics.md about the changed output format.

Thanks

docs/performance-metrics.md doesn't have this schema defined currently. It just says:

| results | A list of metrics |

But the schema of results is not mentioned there.

performance-metrics/src/main.rs

weltling · 2026-03-26T13:20:54Z

performance-metrics/src/main.rs

+                if continue_on_failure {
+                    eprintln!("Test '{}' failed: '{e:?}'. Continuing.", test.name);
+                    has_failure = true;
+                    metrics_report.results.push(PerformanceTestResult {


I'd suggest to unify PerformanceTestResult creation through a helper/constructor, so success/failure status and fields stay consistent and future changes don't diverge.

Thanks

I'm unsure how a constructor/helper would be useful here. Could you please elaborate?

With this change, success and failure construct PerformanceTestResult independently, duplicating the field list and status logic. A helper would centralize that, so future field additions only need updating in one place.

Thanks

Please check now.

LGTM, thanks for addressing this. One could tighten it further with impl Default or alike but given both variants here are on the same page, it's obvious enough in case it's changing.

Thanks

liuw · 2026-03-27T19:02:47Z

It looks like we're trying to replicate what cargo nextest already does.

It should be possible to convert the performance metric tests to use the same framework.

To be clear, I'm not asking you to stop this PR. Switching to cargo nextest is a medium to long term thing.

anirudhrb · 2026-03-30T12:05:47Z

It looks like we're trying to replicate what cargo nextest already does.

It should be possible to convert the performance metric tests to use the same framework.

Yeah, moving these tests to cargo nextest would be great. We wouldn't have to maintain our own test harness, and we'll get a lot of features automatically.

To be clear, I'm not asking you to stop this PR. Switching to cargo nextest is a medium to long term thing.

Ack. I'll continue with this PR for the short term.

russell-islam · 2026-04-03T19:55:07Z

performance-metrics/src/main.rs

+            std_dev,
+            max,
+            min,
+            status: "PASSED".to_string(),


Why can't we use an enum here?

changed it to enum now.

russell-islam · 2026-04-03T19:55:48Z

performance-metrics/src/main.rs

+                .long("continue-on-failure")
+                .help("Continue running remaining tests after a test failure")
+                .num_args(0)
+                .action(ArgAction::SetTrue)


What is the default?

The default is false. ArgAction::SetTrue in clap inherently defaults to false - absent flag means false, passing --continue-on-failure flips it to true.

Add a --continue-on-failure CLI flag that allows the test harness to continue executing remaining tests after encountering a failure, instead of aborting immediately. When set, failed tests are recorded with zeroed metrics and a "FAILED" status, the report file is always generated, and the process exits with a non-zero code if any test failed. Without the flag, the existing fail-fast behavior is preserved. Also add a "status" field ("PASSED"/"FAILED") to PerformanceTestResult so report consumers can distinguish successful tests from failed ones. Signed-off-by: Anirudh Rayabharam <anrayabh@microsoft.com>

anirudhrb requested a review from a team as a code owner March 26, 2026 10:30

weltling reviewed Mar 26, 2026

View reviewed changes

performance-metrics/src/main.rs Outdated Show resolved Hide resolved

weltling reviewed Mar 26, 2026

View reviewed changes

anirudhrb force-pushed the perf_metrics_harness_improvements branch from 7656e98 to 5688819 Compare March 30, 2026 12:32

rbradford force-pushed the perf_metrics_harness_improvements branch from 5688819 to 849f729 Compare April 1, 2026 15:07

anirudhrb force-pushed the perf_metrics_harness_improvements branch from 849f729 to 1ce8c5c Compare April 3, 2026 04:16

weltling approved these changes Apr 3, 2026

View reviewed changes

russell-islam reviewed Apr 3, 2026

View reviewed changes

anirudhrb force-pushed the perf_metrics_harness_improvements branch from 1ce8c5c to e225eea Compare April 6, 2026 12:01

Conversation

anirudhrb commented Mar 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

weltling Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liuw commented Mar 27, 2026

Uh oh!

anirudhrb commented Mar 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anirudhrb Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

weltling Mar 26, 2026 •

edited

Loading

anirudhrb Apr 6, 2026 •

edited

Loading