feat: enrich FailureInfo with FailureAssessment and mock mismatch data by AkashKumar7902 · Pull Request #3996 · keploy/keploy

AkashKumar7902 · 2026-04-02T06:07:41Z

Describe the changes that are made

Add RejectionReason and NoiseFailureReason typed enums to pkg/models/testrun.go for classifying why test cases are unreplayable during autoreplay
Extend FailureInfo struct with Assessment *FailureAssessment and MockMismatch *MockMismatchInfo fields
Add MockMismatchMock and MockMismatchInfo structs to carry expected vs actual mock metadata (name + kind)
Store FailureAssessment in HTTP matcher (pkg/matcher/http/match.go) and gRPC matcher (pkg/matcher/grpc/match.go) instead of discarding it after extracting Risk/Category
Populate MockMismatchInfo on TestResult for obsolete test cases in pkg/service/replay/replay.go

Links & References

Closes: NA

🔗 Related PRs

api-server: adds TestCaseFailureDetails struct using these enums
k8s-proxy: builds failure details per rejected test case using enriched TestResult
enterprise-ui: displays unreplayable reason details on recordings page

🐞 Related Issues

NA

📄 Related Documents

NA

What type of PR is this? (check all applicable)

Added e2e test pipeline?

👍 yes
🙅 no, because they aren't needed

Added comments for hard-to-understand areas?

🙅 no, because the code is self-explanatory

Added to documentation?

🙅 no documentation needed

Are there any sample code or steps to test the changes?

👍 yes, mentioned below

Deploy all 4 PRs (keploy → api-server → k8s-proxy → enterprise-ui), then:

Record test cases against any app
Trigger autoreplay
Check that unreplayable test cases have enriched FailureInfo with Assessment and MockMismatch data

Self Review done?

✅ yes
❌ no, because I need help

Add RejectionReason and NoiseFailureReason typed enums for classifying why test cases are unreplayable during autoreplay. Extend FailureInfo to carry the full FailureAssessment (AddedFields, RemovedFields, TypeChanges, ValueChanges, Reasons) and MockMismatchInfo (expected vs actual mocks with name + kind). Store FailureAssessment in HTTP and gRPC matchers instead of discarding it after extracting Risk/Category. Populate MockMismatchInfo on TestResult for obsolete test cases with rich mock metadata.

kilo-code-bot · 2026-04-02T06:09:01Z

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Overview

This PR adds infrastructure for enriching FailureInfo with detailed failure assessment and mock mismatch data. The implementation is clean and follows existing patterns in the codebase.

Key observations:

The new RejectionReason and NoiseFailureReason enums are defined but intentionally unused in this PR - per the PR description, they're meant to be consumed by related PRs in api-server and k8s-proxy
MockMismatchInfo correctly filters out DNS mocks (which have non-deterministic resolution order) before comparing expected vs actual mocks
Both HTTP and gRPC matchers consistently store bodyAssessment from ComputeFailureAssessmentJSON
The assessment is only populated when JSON parsing succeeds and the assessment is non-nil

Files Reviewed (4 files)

pkg/models/testrun.go - Added enums and extended structs
pkg/matcher/http/match.go - Store body assessment in FailureInfo
pkg/matcher/grpc/match.go - Store body assessment in FailureInfo
pkg/service/replay/replay.go - Populate MockMismatchInfo for obsolete test cases

_{Reviewed by claude-opus-4.5 · 295,791 tokens}

Copilot

Pull request overview

This PR enriches replay failure reporting by carrying structured failure assessment details (risk/category plus detailed assessment data) through HTTP/gRPC matchers and by attaching mock mismatch metadata for obsolete test cases, enabling downstream components (API/UI) to explain why a test is unreplayable.

Changes:

Extend models.FailureInfo with Assessment and MockMismatch plus new enums for rejection/noise failure reasons.
Preserve and propagate JSON body FailureAssessment from HTTP and gRPC matchers into FailureInfo.
Populate MockMismatchInfo on obsolete test results during replay when mock-set mismatch is detected.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
pkg/models/testrun.go	Adds new failure classification enums and extends `FailureInfo` with assessment and mock mismatch metadata models.
pkg/matcher/http/match.go	Keeps the computed JSON `FailureAssessment` instead of discarding it after extracting risk/category.
pkg/matcher/grpc/match.go	Same as HTTP matcher: propagates JSON `FailureAssessment` into `FailureInfo`.
pkg/service/replay/replay.go	Copies matcher assessment into stored test results and attaches expected vs actual mock metadata for obsolete test cases.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-04-02T06:12:49Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.64ms	3.34ms	4.95ms	100.00	0.00%	✅ PASS
2	2.7ms	3.47ms	5.12ms	100.00	0.00%	✅ PASS
3	3.09ms	4.4ms	6.44ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

…le tests - Add ConsumedMock and UnmatchedCall structs to FailureInfo on TestResult - Populate consumed mocks and unmatched calls for failed test cases - Add GetMockErrors HTTP API endpoint on agent for remote error transport - Add GetFailuresForTestCase to TestFailureStore for per-test-case lookup - Fix HTTP outgoing mock recording not persisting in non-synchronous mode - Store FailureAssessment in HTTP/gRPC matchers instead of discarding - Add RejectionReason and NoiseFailureReason enums - Extend FailureInfo with Assessment and MockMismatchInfo fields

github-actions · 2026-04-05T19:55:18Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.76ms	3.56ms	5.27ms	100.02	0.00%	✅ PASS
2	2.65ms	3.38ms	4.96ms	100.02	0.00%	✅ PASS
3	N/A	N/A	N/A	N/A	N/A	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

github-actions · 2026-04-05T20:02:11Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	3.21ms	4.03ms	5.79ms	100.02	0.00%	✅ PASS
2	3.09ms	3.89ms	5.91ms	100.00	0.00%	✅ PASS
3	3.43ms	5.04ms	7.89ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

- Rename ConsumedMock to MatchedCall with fields: MockName, Protocol, Summary - Rename FailureInfo.ConsumedMocks to FailureInfo.MatchedCalls - Add MockSummaryFromSpec() for protocol-generic summaries (HTTP, DNS, MongoDB, MySQL, PostgreSQL, gRPC, Redis, Generic) - Add PreserveFailedMocks config flag to skip mock pruning when tests fail - Use mock registry lookup to build summaries for matched calls

- Normalize expected mock Kind from mockKindByName when empty - Remove accidental blank line in mockmanager.go - Restore accidentally deleted keploy.sh

github-actions · 2026-04-06T09:24:37Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.42ms	3.06ms	4.63ms	100.03	0.00%	✅ PASS
2	2.36ms	2.97ms	4.31ms	100.02	0.00%	✅ PASS
3	2.58ms	3.7ms	5.57ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…optimization, race fix

github-actions · 2026-04-06T09:36:53Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.72ms	3.49ms	5.21ms	100.00	0.00%	✅ PASS
2	2.58ms	3.28ms	4.59ms	100.02	0.00%	✅ PASS
3	2.64ms	3.74ms	6.02ms	100.03	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…iluresForTestCase

github-actions · 2026-04-06T09:55:16Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.8ms	3.61ms	5.19ms	100.00	0.00%	✅ PASS
2	2.75ms	3.64ms	5.21ms	100.02	0.00%	✅ PASS
3	2.9ms	4.31ms	8.15ms	100.00	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…close error log

github-actions · 2026-04-06T10:17:14Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.62ms	3.36ms	4.9ms	100.02	0.00%	✅ PASS
2	2.58ms	3.25ms	4.83ms	100.02	0.00%	✅ PASS
3	2.71ms	4.03ms	7.07ms	100.00	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

In synchronous mode, always route HTTP mocks through the sync manager for correct time-window association. In non-synchronous mode (k8s-proxy), send directly to the mocks channel to ensure persistence.

github-actions · 2026-04-06T10:28:39Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	3.04ms	3.97ms	5.46ms	100.02	0.00%	✅ PASS
2	2.84ms	3.63ms	4.83ms	100.00	0.00%	✅ PASS
3	2.84ms	3.65ms	4.96ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-04-06T10:40:37Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.75ms	3.48ms	5.1ms	100.00	0.00%	✅ PASS
2	2.66ms	3.39ms	5.21ms	100.00	0.00%	✅ PASS
3	2.81ms	4.17ms	6.93ms	100.01	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

When monitorProxyErrors receives context cancellation, Go's select may pick ctx.Done() over pending errors in the channel. This caused mock-not-found errors to be lost, resulting in empty UnmatchedCalls. Now drains remaining errors before returning.

github-actions · 2026-04-06T10:58:57Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.61ms	3.3ms	4.8ms	100.00	0.00%	✅ PASS
2	2.61ms	3.33ms	4.78ms	100.00	0.00%	✅ PASS
3	2.73ms	4.03ms	6.61ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Logs at key points to trace where mock-not-found errors are lost: - monitorProxyErrors: when errors arrive and when ctx.Done fires - GetFailuresForTestCase: all stored failures and match results - replay.go: channel path count, GetMockErrors fallback, final total

This reverts commit d5f9ad7.

This reverts commit df52a0d.

github-actions · 2026-04-06T11:07:58Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.67ms	3.4ms	5.1ms	100.02	0.00%	✅ PASS
2	2.57ms	3.3ms	4.56ms	100.00	0.00%	✅ PASS
3	2.72ms	4ms	6.19ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

…in GetFailuresForTestCase" This reverts commit 0888c38.

github-actions · 2026-04-06T13:08:43Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.8ms	3.65ms	5.33ms	100.02	0.00%	✅ PASS
2	2.69ms	3.54ms	5.06ms	100.00	0.00%	✅ PASS
3	3.03ms	4.34ms	6.29ms	100.00	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sarthak160

LGTM

Copilot AI review requested due to automatic review settings April 2, 2026 06:07

AkashKumar7902 requested review from Sarthak160 and gouravkrosx as code owners April 2, 2026 06:07

Copilot started reviewing on behalf of AkashKumar7902 April 2, 2026 06:08

View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

Comment thread pkg/matcher/http/match.go Resolved

Comment thread pkg/matcher/http/match.go Resolved

Comment thread pkg/matcher/grpc/match.go Resolved

Comment thread pkg/service/replay/replay.go Resolved

feat: enrich ConsumedMock with summary from mock registry

51b3b64

AkashKumar7902 force-pushed the feat/rejection-reason-enrichment branch from 21c5661 to 51b3b64 Compare April 5, 2026 19:56

AkashKumar7902 added 3 commits April 6, 2026 12:13

refactor: address PR review comments

1904cce

- Normalize expected mock Kind from mockKindByName when empty - Remove accidental blank line in mockmanager.go - Restore accidentally deleted keploy.sh

Merge branch 'main' into feat/rejection-reason-enrichment

fe0bb39

AkashKumar7902 requested a review from Copilot April 6, 2026 09:19

Copilot started reviewing on behalf of AkashKumar7902 April 6, 2026 09:20

View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

fix: address copilot review — render ordering, nil guard, mockLookup …

0225170

…optimization, race fix

AkashKumar7902 requested a review from Copilot April 6, 2026 09:35

Copilot started reviewing on behalf of AkashKumar7902 April 6, 2026 09:37

View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

Comment thread pkg/service/replay/replay.go Resolved

Comment thread pkg/service/replay/utils.go Resolved

fix: drain consumed mocks for all test cases, defensive copy in GetFa…

0888c38

…iluresForTestCase

AkashKumar7902 requested a review from Copilot April 6, 2026 09:51

Copilot started reviewing on behalf of AkashKumar7902 April 6, 2026 09:52

View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

Comment thread pkg/agent/proxy/integrations/http/http.go Resolved

Comment thread pkg/platform/http/agent.go Resolved

fix: update comment for AddMock behavior, add actionable guidance to …

f21d259

…close error log

AkashKumar7902 requested a review from Copilot April 6, 2026 10:13

Copilot started reviewing on behalf of AkashKumar7902 April 6, 2026 10:14

View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

Comment thread pkg/agent/proxy/integrations/http/http.go Resolved

fix: gate syncMock usage on opts.Synchronous instead of firstReqSeen

f42d0e8

In synchronous mode, always route HTTP mocks through the sync manager for correct time-window association. In non-synchronous mode (k8s-proxy), send directly to the mocks channel to ensure persistence.

AkashKumar7902 requested a review from Copilot April 6, 2026 10:24

Copilot started reviewing on behalf of AkashKumar7902 April 6, 2026 10:28

View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

Comment thread pkg/models/testrun.go Resolved

Comment thread pkg/service/replay/replay.go Resolved

feat: propagate NextSteps from MockMismatchReport to UnmatchedCall

4cdd717

AkashKumar7902 requested a review from Copilot April 6, 2026 10:37

Copilot started reviewing on behalf of AkashKumar7902 April 6, 2026 10:38

View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

Comment thread pkg/service/replay/replay.go Resolved

Comment thread pkg/agent/proxy/proxy.go Resolved

AkashKumar7902 added 3 commits April 6, 2026 16:32

Revert "debug: add diagnostic logs for unmatched calls flow"

6c045b4

This reverts commit d5f9ad7.

Revert "fix: drain remaining proxy errors on context cancellation"

1feb9f2

This reverts commit df52a0d.

Revert "fix: drain consumed mocks for all test cases, defensive copy …

1efb7b5

…in GetFailuresForTestCase" This reverts commit 0888c38.

AkashKumar7902 requested a review from Copilot April 6, 2026 13:11

Copilot started reviewing on behalf of AkashKumar7902 April 6, 2026 13:12

View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

Comment thread pkg/service/replay/replay.go Resolved

Sarthak160 approved these changes Apr 6, 2026

View reviewed changes

AkashKumar7902 merged commit 4d82f3f into main Apr 6, 2026
132 of 134 checks passed

AkashKumar7902 deleted the feat/rejection-reason-enrichment branch April 6, 2026 13:23

github-actions Bot locked and limited conversation to collaborators Apr 6, 2026

Conversation

AkashKumar7902 commented Apr 2, 2026

Describe the changes that are made

Links & References

🔗 Related PRs

🐞 Related Issues

📄 Related Documents

What type of PR is this? (check all applicable)

Added e2e test pipeline?

Added comments for hard-to-understand areas?

Added to documentation?

Are there any sample code or steps to test the changes?

Self Review done?

Uh oh!

kilo-code-bot Bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Summary

Overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 2, 2026

🚀 Keploy Performance Test Results

Uh oh!

github-actions Bot commented Apr 5, 2026

🚀 Keploy Performance Test Results

Uh oh!

github-actions Bot commented Apr 5, 2026

🚀 Keploy Performance Test Results

Uh oh!

github-actions Bot commented Apr 6, 2026

🚀 Keploy Performance Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 6, 2026

🚀 Keploy Performance Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 6, 2026

🚀 Keploy Performance Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 6, 2026

🚀 Keploy Performance Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions Bot commented Apr 6, 2026

kilo-code-bot Bot commented Apr 2, 2026 •

edited

Loading