feat(replay): add TestCaseMutator hook for pre-replay test case mutations by Vedanshi27vishu · Pull Request #4238 · keploy/keploy

Vedanshi27vishu · 2026-06-02T12:33:21Z

Summary

Adds an optional TestCaseMutator interface to the replay service so that enterprise/custom implementations can mutate test cases (e.g. decrypt secrets, inject headers) immediately before each test case is replayed — without modifying the core TestHooks interface.

Why a separate interface?

Extending TestHooks directly would be a compile-time breaking change for any external hook implementations. TestCaseMutator is detected via type assertion in RunTestSet, so existing implementations that do not implement it continue to work unchanged.

Changes

pkg/service/replay/service.go

Added TestCaseMutator interface with a single method:

BeforeTestCaseRun(ctx context.Context, tc *models.TestCase, testSetID string) error

pkg/service/replay/replay.go

Call BeforeTestCaseRun via type assertion before ReplaceBaseURL and SimulateRequest
Gated with replay == 0 to prevent double-mutation on retry passes
Covers both normal and streaming test case paths

How decryption fits in

The actual AES-256-GCM + KMS decryption of ENC: values lives in the enterprise repo (private). Enterprise implements TestCaseMutator and wires it in at startup. This PR only adds the OSS hook point.

Merge order

Must merge before enterprise and k8s-proxy PRs.

github-actions · 2026-06-02T12:33:37Z

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I hereby sign the CLA

_{You can retrigger this bot by commenting recheck in this Pull Request}

github-actions · 2026-06-02T12:33:38Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions

Thank you and congratulations 🎉 for opening your very first pull request in keploy

github-actions · 2026-06-02T12:34:44Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

Copilot

Pull request overview

Adds replay-time support for decrypting ENC:<keyId>:<base64> request values (recorded with encryption_protection.enabled=true) by fetching AES-256 keys from the api-server KMS, so replayed requests send real secret values rather than the raw ENC: strings.

Changes:

Introduces a per-run KMS key cache + AES-256-GCM decryption utility for ENC:-prefixed values.
Wires request decryption into the HTTP replay path (including deferred streaming HTTP tests) before SimulateRequest.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 9 comments.

File	Description
pkg/service/replay/decrypt.go	New KMS key fetch/cache and `ENC:` decryption logic for HTTP request fields.
pkg/service/replay/replay.go	Integrates decryption into `RunTestSet` before request simulation (standard + streaming HTTP).

Comments suppressed due to low confidence (2)

pkg/service/replay/replay.go:1877

The streaming path also decrypts the request in-place and then persists tc.HTTPReq into the report (models.TestResult.Req), which can re-expose secrets that were stored as ENC: in the recording. Decrypt only for the outgoing request and restore the original encrypted values before writing results.

						MockPath:     filepath.Join(r.config.Path, testSetID, "mocks.yaml"),
						Noise:        testCase.Noise,
						Result:       *testResult,
						TimeTaken:    time.Since(started).String(),

pkg/service/replay/replay.go:1455

Decrypting in-place mutates testCase.HTTPReq and later persists the decrypted header/params/body into models.TestResult.Req, which can leak secrets that were intentionally stored as ENC: in the recording. Consider decrypting only for the outgoing simulation and then restoring the original encrypted values before building reports/results.


	var activeTestCases []*models.TestCase
	var streamingTests []streamingTest
	for _, testCase := range testCases {
		if _, ok := selectedTests[testCase.Name]; !ok && len(selectedTests) != 0 {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-06-03T08:16:12Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions · 2026-06-03T10:38:14Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions · 2026-06-03T10:43:23Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.7ms	3.4ms	4.86ms	100.00	0.00%	✅ PASS
2	2.63ms	3.33ms	4.83ms	100.00	0.00%	✅ PASS
3	2.68ms	3.6ms	5.2ms	100.00	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Signed-off-by: Vedanshi Aggarwal <vedanshi27vishu@gmail.com>

…r, Debug logs, unit tests Signed-off-by: Vedanshi Aggarwal <vedanshi27vishu@gmail.com>

github-actions · 2026-06-03T11:12:25Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions · 2026-06-03T11:16:11Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.64ms	3.37ms	5.12ms	100.02	0.00%	✅ PASS
2	2.6ms	3.31ms	4.92ms	100.02	0.00%	✅ PASS
3	2.66ms	3.65ms	5.56ms	100.00	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

github-actions · 2026-06-03T11:17:58Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions · 2026-06-03T11:21:21Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	3.11ms	3.9ms	5.37ms	100.00	0.00%	✅ PASS
2	2.99ms	3.76ms	5.11ms	100.00	0.00%	✅ PASS
3	2.92ms	3.73ms	5.15ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

github-actions · 2026-06-04T11:11:35Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions · 2026-06-04T11:16:14Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.76ms	3.53ms	5.12ms	100.02	0.00%	✅ PASS
2	2.81ms	3.56ms	5.08ms	100.02	0.00%	✅ PASS
3	3.07ms	4.04ms	5.93ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Replace the inline kmsCache + decrypt.go approach with a minimal BeforeTestCaseRun hook in the TestHooks interface. OSS keploy now only calls the hook; actual AES-256-GCM + KMS decryption lives in enterprise's DecryptingTestHooks implementation. - Add BeforeTestCaseRun(ctx, tc, testSetID) to TestHooks interface - Add no-op default in Hooks struct - Call hook in replay.go before SimulateRequest (both normal and streaming) - Remove decrypt.go, decrypt_test.go (moved to enterprise) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Vedanshi Aggarwal <vedanshi27vishu@gmail.com>

github-actions · 2026-06-05T09:24:44Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions · 2026-06-05T09:28:22Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.69ms	3.51ms	4.96ms	100.00	0.00%	✅ PASS
2	2.63ms	3.37ms	4.92ms	100.00	0.00%	✅ PASS
3	2.6ms	3.37ms	4.76ms	100.00	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Move BeforeTestCaseRun hook call to execute before ReplaceBaseURL so that encrypted URL query params are decrypted before the base URL replacement parses them. Decryption must be the first operation on the test case — before any field-level processing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Vedanshi Aggarwal <vedanshi27vishu@gmail.com>

github-actions · 2026-06-05T09:41:04Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

- Fix log message: "unmodified test case" → "skipping pre-replay mutation" (tc may be partially mutated before error returns) - Gate ReplaceBaseURL with replay==0 to prevent path.Join doubling the base-path prefix on retry passes - Add comment on streaming path explaining why replay==0 guard is not needed (Phase 2 has no retry loop) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Vedanshi Aggarwal <vedanshi27vishu@gmail.com>

github-actions · 2026-06-05T11:23:08Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions · 2026-06-05T11:26:41Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.57ms	3.15ms	4.71ms	100.00	0.00%	✅ PASS
2	2.53ms	3.05ms	4.46ms	100.00	0.00%	✅ PASS
3	2.54ms	3.13ms	4.38ms	100.03	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

- Move exitLoopChan check before BeforeTestCaseRun so we don't mutate test cases that will never execute when an exit signal is pending - Move streaming BeforeTestCaseRun before effectiveStreamMockWindow so any timestamps decrypted by the mutator feed into the mock filter window - Neutralise hook error log message — 'skipping' implied rollback but tc is passed by pointer and may be partially mutated before the error Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-05T11:49:19Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

+					if err := mutator.BeforeTestCaseRun(runTestSetCtx, testCase, testSetID); err != nil {
+						utils.LogError(r.logger, err, "BeforeTestCaseRun hook failed; replay continues with test case in current state",
+							zap.String("testcase", testCase.Name),
+							zap.String("next_step", "check hook implementation or api-server connectivity"))


+				if err := mutator.BeforeTestCaseRun(runTestSetCtx, tc, testSetID); err != nil {
+					utils.LogError(r.logger, err, "BeforeTestCaseRun hook failed; replay continues with test case in current state",
+						zap.String("testcase", tc.Name),
+						zap.String("next_step", "check hook implementation or api-server connectivity"))


github-actions · 2026-06-05T11:52:56Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.79ms	3.61ms	4.96ms	100.00	0.00%	✅ PASS
2	2.69ms	3.5ms	5.01ms	100.02	0.00%	✅ PASS
3	2.78ms	3.71ms	5.49ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

next_step was hard-coded to "api-server connectivity" but TestCaseMutator is a general-purpose hook — the hint now lists common dependency types (KMS, auth, network) so it stays accurate for all custom implementations. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-05T11:54:53Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

github-actions · 2026-06-05T11:58:55Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.69ms	3.37ms	5ms	100.03	0.00%	✅ PASS
2	2.6ms	3.25ms	4.68ms	100.02	0.00%	✅ PASS
3	2.65ms	3.48ms	5.11ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

The comment said "before any in-place mutations" but IsLast is already set earlier in the loop — narrowed to "before the hook and URL mutations" which is what the gate actually protects. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-05T11:59:47Z

The CLA check failed. Please ensure you have:

Signed the CLA by commenting 'I have read the CLA Document and I hereby sign the CLA.'
Used the correct email address in your commits (matches the one you used to sign the CLA).

After fixing these issues, comment 'recheck' to trigger the workflow again.

github-actions · 2026-06-05T12:03:50Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.94ms	3.77ms	5.21ms	100.02	0.00%	✅ PASS
2	2.8ms	3.58ms	5.24ms	100.02	0.00%	✅ PASS
3	2.88ms	3.8ms	6.34ms	100.02	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings June 2, 2026 12:33

Vedanshi27vishu requested a review from gouravkrosx as a code owner June 2, 2026 12:33

Copilot started reviewing on behalf of Vedanshi27vishu June 2, 2026 12:33 View session

github-actions Bot reviewed Jun 2, 2026

View reviewed changes

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Vedanshi27vishu and others added 5 commits June 3, 2026 16:28

completed

5b266eb

Signed-off-by: Vedanshi Aggarwal <vedanshi27vishu@gmail.com>

Merge branch 'main' into feat/secret-encryption

a885429

Merge branch 'main' into feat/secret-encryption

9c77dae

fix(replay): remove tokenProvider reference from OSS Replayer

be881f8

Signed-off-by: Vedanshi Aggarwal <vedanshi27vishu@gmail.com>

fix(replay): address Copilot review — url.PathEscape, io.ReadAll erro…

f92d2d3

…r, Debug logs, unit tests Signed-off-by: Vedanshi Aggarwal <vedanshi27vishu@gmail.com>

Vedanshi27vishu force-pushed the feat/secret-encryption branch from e19df50 to f92d2d3 Compare June 3, 2026 11:12

Merge branch 'main' into feat/secret-encryption

492ac23

Merge branch 'main' into feat/secret-encryption

a8b14eb

Vedanshi27vishu requested a review from Copilot June 5, 2026 11:00

Copilot started reviewing on behalf of Vedanshi27vishu June 5, 2026 11:01 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread pkg/service/replay/replay.go Outdated

Comment thread pkg/service/replay/replay.go Outdated

Comment thread pkg/service/replay/replay.go Outdated

Vedanshi27vishu requested a review from Copilot June 5, 2026 11:23

Copilot started reviewing on behalf of Vedanshi27vishu June 5, 2026 11:23 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread pkg/service/replay/replay.go

Vedanshi27vishu requested a review from Copilot June 5, 2026 11:39

Copilot started reviewing on behalf of Vedanshi27vishu June 5, 2026 11:39 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread pkg/service/replay/replay.go

Comment thread pkg/service/replay/replay.go

Comment thread pkg/service/replay/replay.go Outdated

Vedanshi27vishu requested a review from Copilot June 5, 2026 11:50

Copilot started reviewing on behalf of Vedanshi27vishu June 5, 2026 11:50 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Vedanshi27vishu requested a review from Copilot June 5, 2026 11:55

Copilot started reviewing on behalf of Vedanshi27vishu June 5, 2026 11:55 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread pkg/service/replay/replay.go Outdated

Comment thread pkg/service/replay/replay.go

Vedanshi27vishu requested a review from Copilot June 5, 2026 11:59

Copilot started reviewing on behalf of Vedanshi27vishu June 5, 2026 12:00 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Uh oh!

Conversation

Vedanshi27vishu commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why a separate interface?

Changes

How decryption fits in

Merge order

Uh oh!

github-actions Bot commented Jun 2, 2026

Uh oh!

github-actions Bot commented Jun 2, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

🚀 Keploy Performance Test Results

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

🚀 Keploy Performance Test Results

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

🚀 Keploy Performance Test Results

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

🚀 Keploy Performance Test Results

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

🚀 Keploy Performance Test Results

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

🚀 Keploy Performance Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Vedanshi27vishu commented Jun 2, 2026 •

edited

Loading