feat: keep docker-compose content in memory to prevent env var exposure by ayush3160 · Pull Request #4022 · keploy/keploy

ayush3160 · 2026-04-08T01:03:53Z

Summary

Add InMemoryCompose []byte to config.Config and models.SetupOptions to thread in-memory compose content from the replayer down to App without touching the filesystem.
Add FindContainerInCompose and MarshalCompose to docker.Client interface (and Impl) for operating on in-memory Compose structs.
App.SetupCompose branches on opts.InMemoryCompose: when set, it parses the YAML in memory, injects the keploy-agent service, re-serialises to bytes, and stores the result in App.composeContent.
App.run pipes composeContent via cmd.Stdin using docker compose -f -; App.composeDown does the same for teardown.
ExecuteCommand gains a stdin []byte parameter; when non-nil it sets cmd.Stdin and skips PTY (PTY cannot coexist with an explicit stdin reader).
Add ensureInMemoryComposeFlags helper that rewrites -f <path> to -f - and injects --abort-on-container-exit/--exit-code-from flags.

Motivation

Addresses keploy/enterprise#1827 — the enterprise cloud replay command was writing a docker-compose-keploy.yaml file containing K8s secrets and user-supplied env vars to the working directory. This PR provides the OSS-layer plumbing needed by the companion enterprise PR to keep all compose content in process memory.

Test plan

go vet ./... passes on both utils/ and pkg/ packages
Docker Compose replay with --envs flag: verify no docker-compose-*.yaml file is created in the working directory
docker compose down teardown completes successfully after an in-memory run
Non-cloud replay (file-based compose path) is unaffected — composeContent is nil so the old file-based path runs unchanged
PTY behaviour unaffected for normal (non-in-memory) docker compose runs

🤖 Generated with Claude Code

When the enterprise cloud command generates a docker-compose config from Kubernetes resources (Deployments, ConfigMaps, Secrets + user --envs), the resulting YAML containing secrets and tokens is now held entirely in process memory rather than written to docker-compose-keploy.yaml on disk. Changes: - Add InMemoryCompose []byte to config.Config and models.SetupOptions so compose content can be threaded from the replayer down to App without touching the filesystem. - Add FindContainerInCompose and MarshalCompose to the docker.Client interface (and Impl) for operating on in-memory Compose structs. - App.SetupCompose branches on opts.InMemoryCompose: when set it parses the YAML in memory, injects the keploy-agent service, re-serialises to bytes, and stores the result in App.composeContent. - App.run pipes composeContent via cmd.Stdin using "docker compose -f -". - App.composeDown pipes composeContent via Stdin for teardown too. - ExecuteCommand gains a stdin []byte parameter; when non-nil it sets cmd.Stdin and skips PTY (PTY cannot coexist with an explicit stdin). - Add ensureInMemoryComposeFlags helper that rewrites "-f <path>" to "-f -" and injects --abort-on-container-exit/--exit-code-from flags. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-04-08T01:09:42Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.7ms	3.42ms	5.21ms	100.02	0.00%	✅ PASS
2	2.62ms	3.34ms	5.35ms	100.02	0.00%	✅ PASS
3	2.75ms	4.02ms	6.05ms	100.01	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Adds an in-memory Docker Compose execution path so replay can inject the keploy-agent service and run docker compose via stdin (-f -) without writing potentially sensitive compose YAML (env vars/secrets) to disk.

Changes:

Thread InMemoryCompose []byte through config and setup options into the app setup flow.
Extend the Docker client with in-memory compose helpers (FindContainerInCompose, MarshalCompose) and use them in App.SetupCompose.
Support piping stdin into command execution and disable PTY when stdin content is provided.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
utils/signal_windows.go	Adds optional stdin piping support for Windows command execution.
utils/signal_others.go	Adds optional stdin piping support and skips PTY when stdin is provided.
pkg/service/replay/replay.go	Threads `InMemoryCompose` into setup; updates `ExecuteCommand` call signature.
pkg/platform/docker/service.go	Extends Docker client interface with in-memory compose helpers.
pkg/platform/docker/docker.go	Implements `MarshalCompose` and `FindContainerInCompose`.
pkg/models/instrument.go	Adds `InMemoryCompose` to `models.SetupOptions`.
pkg/client/app/util.go	Adds `ensureInMemoryComposeFlags` to rewrite compose CLI for stdin usage.
pkg/client/app/app.go	Adds in-memory compose setup path and stdin-based `up`/`down` execution plumbing.
config/config.go	Adds `InMemoryCompose` to config for plumbing (non-serialized).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-08T01:10:18Z

+	// Replace any existing "-f <path>" with "-f -" so docker compose reads from stdin.
+	pattern := `(-f\s+("[^"]+"|'[^']+'|\S+))`
+	re := regexp.MustCompile(pattern)
+	if re.MatchString(appCmd) {
+		appCmd = re.ReplaceAllString(appCmd, "-f -")
+	} else {


ensureInMemoryComposeFlags uses a regex ReplaceAllString that will rewrite every "-f " occurrence to "-f -". For commands that specify multiple compose files (e.g. "-f a.yml -f b.yml"), this produces multiple "-f -" flags, which makes docker compose try to read stdin multiple times and can fail/hang. Consider tokenizing args and collapsing all -f/--file flags into a single "-f -" (or remove all existing file flags and inject exactly one).

Copilot · 2026-04-08T01:10:18Z

+	switch {
+	case len(a.composeContent) > 0:
+		// In-memory mode: pipe compose YAML via stdin, no file on disk.
+		a.logger.Debug("Running docker compose down using in-memory compose content")
+		downCmd = exec.Command("docker", "compose", "-f", "-", "down")
+		downCmd.Stdin = bytes.NewReader(a.composeContent)
+	case a.composeFile != "":
+		a.logger.Debug("Running docker compose down to clean up containers and networks",
+			zap.String("composeFile", a.composeFile))
+		downCmd = exec.Command("docker", "compose", "-f", a.composeFile, "down")


In in-memory mode, composeDown runs docker compose -f - down without preserving any project-scoping flags from the original a.cmd (e.g. -p/--project-name or --project-directory). If the user’s compose command used a non-default project name, this teardown can target the wrong project and leave containers/networks running. Consider extracting the project name/dir from a.cmd (or using the com.docker.compose.project label via composeProjectLabel) and passing the same flags to the down command.

- Collapse all -f/--file flags into a single -f - in ensureInMemoryComposeFlags to prevent multiple stdin readers when multiple compose files are specified - Preserve project-scoping flags (-p/--project-name, --project-directory) in composeDown so teardown targets the correct docker compose project Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-actions · 2026-04-09T10:56:21Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-09T11:05:59Z

+func ensureInMemoryComposeFlags(appCmd, serviceName string) string {
+	parts := strings.Fields(appCmd)
+
+	// Strip every existing -f/--file flag and its value.
+	cleaned := make([]string, 0, len(parts))
+	for i := 0; i < len(parts); i++ {
+		if (parts[i] == "-f" || parts[i] == "--file") && i+1 < len(parts) {
+			i++ // skip the path argument
+			continue
+		}
+		if strings.HasPrefix(parts[i], "-f=") || strings.HasPrefix(parts[i], "--file=") {
+			continue
+		}
+		cleaned = append(cleaned, parts[i])
+	}
+
+	// Inject a single "-f -" right before "up" (or at the end when "up" is absent).
+	injected := false
+	result := make([]string, 0, len(cleaned)+2)
+	for _, p := range cleaned {
+		if p == "up" && !injected {
+			result = append(result, "-f", "-")
+			injected = true
+		}
+		result = append(result, p)
+	}
+	if !injected {
+		result = append(result, "-f", "-")
+	}
+
+	return ensureComposeExitOnAppFailure(strings.Join(result, " "), serviceName)
+}


ensureInMemoryComposeFlags can generate an invalid docker-compose command when there is no literal "up" token: it appends -f - at the end, which places a global option after the subcommand (e.g., "docker compose ps" becomes "docker compose ps -f -"). Also, the “strip” loop leaves a dangling -f/--file token untouched if it appears without a value (last arg), which can further corrupt the command. Consider (1) inserting -f - before the first compose subcommand token (up/down/run/ps/etc.) rather than appending, and (2) dropping lone -f/--file tokens even when there is no following value.

Copilot · 2026-04-09T11:05:59Z

+	if len(stdin) > 0 {
+		cmd.Stdin = bytes.NewReader(stdin)
+	}


The PR description says PTY is skipped and stdin is set “when non-nil”, but the implementation uses len(stdin) > 0 / len(stdin) == 0. If a caller passes a non-nil but empty slice (e.g., as a sentinel to disable PTY), this code will still take the PTY path and won’t set cmd.Stdin. Aligning behavior with the description would mean checking stdin != nil (or updating the description and ensuring all callers always pass non-empty content).

Copilot · 2026-04-09T11:05:59Z

 	// A PTY gives Docker Compose its own terminal to work with.
-	if cmdType == DockerCompose && isTTY {
+	// Skip PTY when stdin content is provided (in-memory compose mode).
+	if cmdType == DockerCompose && isTTY && len(stdin) == 0 {


The PR description says PTY is skipped and stdin is set “when non-nil”, but the implementation uses len(stdin) > 0 / len(stdin) == 0. If a caller passes a non-nil but empty slice (e.g., as a sentinel to disable PTY), this code will still take the PTY path and won’t set cmd.Stdin. Aligning behavior with the description would mean checking stdin != nil (or updating the description and ensuring all callers always pass non-empty content).

Copilot · 2026-04-09T11:06:00Z

+		args = append(args, extractProjectFlags(a.cmd)...)
+		args = append(args, "down")
+		downCmd = exec.Command("docker", args...)
+		downCmd.Stdin = bytes.NewReader(a.composeContent)


bytes.NewReader is newly used in this file; ensure bytes is imported in pkg/client/app/app.go or this won’t compile. (If bytes is already imported elsewhere in the file, this can be ignored.)

Copilot · 2026-04-09T11:06:00Z

+func extractProjectFlags(cmd string) []string {
+	parts := strings.Fields(cmd)
+	var flags []string
+	for i := 0; i < len(parts); i++ {
+		switch {
+		case (parts[i] == "-p" || parts[i] == "--project-name" || parts[i] == "--project-directory") && i+1 < len(parts):
+			flags = append(flags, parts[i], parts[i+1])
+			i++
+		case strings.HasPrefix(parts[i], "--project-name=") || strings.HasPrefix(parts[i], "--project-directory="):
+			flags = append(flags, parts[i])
+		}
+	}
+	return flags
+}


extractProjectFlags misses a couple of common forms and can fail to preserve project scoping in teardown: it doesn’t handle -p=<name> and it inherits the same quoting/path-with-spaces breakage from strings.Fields. Consider parsing args with a quote-aware lexer and supporting -p=<name> as well, so composeDown reliably targets the same project as the original docker compose run.

- Insert -f - before the first compose subcommand (not just "up") and handle dangling -f/--file tokens without a following value - Use stdin != nil instead of len(stdin) > 0 so a non-nil empty slice still disables PTY as intended by the PR description - Support -p=<name> form in extractProjectFlags for composeDown Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-actions · 2026-04-09T11:23:26Z

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run	P50	P90	P99	RPS	Error Rate	Status
1	2.69ms	3.46ms	4.97ms	100.02	0.00%	✅ PASS
2	2.63ms	3.38ms	4.85ms	100.02	0.00%	✅ PASS
3	2.79ms	3.91ms	6.07ms	100.00	0.00%	✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

charankamarapu

LGTM

Copilot AI review requested due to automatic review settings April 8, 2026 01:03

ayush3160 requested a review from gouravkrosx as a code owner April 8, 2026 01:03

Copilot started reviewing on behalf of ayush3160 April 8, 2026 01:04 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

ayush3160 and others added 2 commits April 9, 2026 16:02

Merge branch 'main' into feat/in-memory-docker-compose

845f888

ayush3160 requested a review from Copilot April 9, 2026 10:58

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Copilot started reviewing on behalf of ayush3160 April 9, 2026 11:17 View session

charankamarapu approved these changes Apr 9, 2026

View reviewed changes

slayerjain merged commit 2baa677 into main Apr 9, 2026
129 checks passed

slayerjain deleted the feat/in-memory-docker-compose branch April 9, 2026 11:58

github-actions Bot locked and limited conversation to collaborators Apr 9, 2026

Uh oh!

Conversation

ayush3160 commented Apr 8, 2026

Summary

Motivation

Test plan

Uh oh!

github-actions Bot commented Apr 8, 2026

🚀 Keploy Performance Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 9, 2026

🚀 Keploy Performance Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 9, 2026

🚀 Keploy Performance Test Results

Uh oh!

charankamarapu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants