feat: add global --json flag for structured output across all commands#4012
feat: add global --json flag for structured output across all commands#4012officialasishkumar merged 6 commits intokeploy:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a global --json mode intended to make Keploy CLI output machine-readable, primarily by emitting JSON for keploy test and keploy report and suppressing the logo to keep stdout clean.
Changes:
- Introduces
utils.JSONWriter+ unit tests to emit JSON to stdout when enabled. - Adds
--jsonas a persistent root flag and wires it into config (Config.JSONOutput), skipping logo printing in JSON mode. - Adds JSON output paths in report generation (
pkg/service/report) and test-run summary printing (pkg/service/replay).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
utils/output.go |
Adds a JSONWriter helper used by commands to emit JSON. |
utils/output_test.go |
Unit tests validating JSONWriter output / disabled behavior / marshal error. |
cli/provider/cmd.go |
Adds global --json flag, reads it into config, suppresses logo when enabled. |
config/config.go |
Adds JSONOutput field to config struct. |
pkg/service/report/report.go |
Emits JSON for DB-backed and file-backed report generation when JSON mode is enabled. |
pkg/service/replay/replay.go |
Emits JSON per test-set summary and suppresses the final text summary in JSON mode. |
Comments suppressed due to low confidence (1)
pkg/service/report/report.go:489
generateReportFromFilereturns the full decoded TestReport as JSON immediately when--jsonis set, bypassing--summaryand--test-case-idsbehavior that is otherwise supported in file mode. JSON mode should honor the same selection flags (e.g., summary-only and/or filtered tests) for consistent CLI semantics.
// Attempt to parse the file into the canonical TestReport struct.
var tr models.TestReport
err = dec.Decode(&tr)
if err == nil && (tr.Name != "" || len(tr.Tests) > 0) {
if r.config.JSONOutput {
return utils.NewJSONWriter(true).Write(tr)
}
// Summary-only
if r.config.Report.Summary {
m := map[string]*models.TestReport{tr.Name: &tr}
return r.printSummary(m)
}
// Test-case filtering
if len(r.config.Report.TestCaseIDs) > 0 {
sel := r.filterTestsByIDs(tr.Tests, r.config.Report.TestCaseIDs)
if len(sel) == 0 {
r.logger.Warn("No matching test-cases found in file", zap.Strings("ids", r.config.Report.TestCaseIDs))
return nil
}
return r.printTests(ctx, sel)
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Debug bool `json:"debug" yaml:"debug" mapstructure:"debug"` | ||
| DisableTele bool `json:"disableTele" yaml:"disableTele" mapstructure:"disableTele"` | ||
| DisableANSI bool `json:"disableANSI" yaml:"disableANSI" mapstructure:"disableANSI"` | ||
| JSONOutput bool `json:"json_output" yaml:"json_output" mapstructure:"json_output"` |
There was a problem hiding this comment.
Config field tags for JSONOutput use json_output/yaml:"json_output", but the rest of Config uses camelCase keys (e.g. disableANSI). This makes config-file/env key naming inconsistent and will prevent jsonOutput/disableANSI-style configs from unmarshalling into JSONOutput. Align the tags with the existing convention (and keep the --json flag aliasing if desired).
| JSONOutput bool `json:"json_output" yaml:"json_output" mapstructure:"json_output"` | |
| JSONOutput bool `json:"jsonOutput" yaml:"jsonOutput" mapstructure:"jsonOutput"` |
| Debug bool `json:"debug" yaml:"debug" mapstructure:"debug"` | ||
| DisableTele bool `json:"disableTele" yaml:"disableTele" mapstructure:"disableTele"` | ||
| DisableANSI bool `json:"disableANSI" yaml:"disableANSI" mapstructure:"disableANSI"` | ||
| JSONOutput bool `json:"json_output" yaml:"json_output" mapstructure:"json_output"` | ||
| InDocker bool `json:"inDocker" yaml:"-" mapstructure:"inDocker"` |
There was a problem hiding this comment.
config/default.go (defaultConfig YAML template) doesn’t include the newly added JSONOutput key, so generated configs won’t surface this option and may silently omit it. Please add the appropriate default entry to the template to keep config generation complete.
| func (c *CmdConfigurator) ValidateFlags(ctx context.Context, cmd *cobra.Command) error { | ||
| jsonOutput, err := cmd.Flags().GetBool("json") | ||
| if err != nil { | ||
| errMsg := "failed to get the json flag" | ||
| utils.LogError(c.logger, err, errMsg) | ||
| return errors.New(errMsg) | ||
| } | ||
| c.cfg.JSONOutput = jsonOutput | ||
|
|
||
| disableAnsi, _ := (cmd.Flags().GetBool("disable-ansi")) | ||
| // Skip printing logo for agent command to avoid duplicate logos in native mode | ||
| if cmd.Name() != "agent" { | ||
| if cmd.Name() != "agent" && !c.cfg.JSONOutput { | ||
| PrintLogo(os.Stdout, disableAnsi) |
There was a problem hiding this comment.
In JSON output mode, logs still go to stdout (logger core is wired to os.Stdout), which will mix non-JSON lines into the structured output and break piping to jq. Consider redirecting logs to stderr (or silencing non-essential logs) when --json is enabled, and also forcing DisableANSI to true in this mode to avoid ANSI escape sequences.
| if len(r.config.Report.TestCaseIDs) > 0 { | ||
| for name, rep := range reports { | ||
| rep.Tests = r.filterTestsByIDs(rep.Tests, r.config.Report.TestCaseIDs) | ||
| rep.Total = len(rep.Tests) | ||
| reports[name] = rep | ||
| } |
There was a problem hiding this comment.
When filtering by TestCaseIDs in JSON mode, only Total is recomputed but Success/Failure/Ignored/Obsolete (and risk counters) remain from the unfiltered report, so the JSON summary fields become inconsistent with the Tests array. Recalculate all derived counters after filtering (or provide a helper that rebuilds the summary from Tests).
| // Define legacy report structure | ||
| type legacy struct { | ||
| Tests []models.TestResult `yaml:"tests"` | ||
| } | ||
|
|
||
| var lg legacy | ||
| dec := yaml.NewDecoder(f) | ||
| err = dec.Decode(&lg) | ||
| if err != nil { | ||
| r.logger.Error("failed to parse report file with legacy parser", zap.String("report_path", reportPath), zap.Error(err)) | ||
| return err | ||
| } | ||
|
|
||
| if r.config.JSONOutput { | ||
| return utils.NewJSONWriter(true).Write(lg) | ||
| } |
There was a problem hiding this comment.
Legacy file parsing JSON output returns the raw legacy struct without applying any of the selection flags (summary/test-case filtering) and without a stable top-level schema. Consider normalizing legacy JSON output to the same shape as the non-legacy report output (or explicitly documenting that legacy JSON differs).
| type JSONWriter struct { | ||
| enabled bool | ||
| } | ||
|
|
||
| func NewJSONWriter(enabled bool) *JSONWriter { | ||
| return &JSONWriter{enabled: enabled} | ||
| } | ||
|
|
||
| func (w *JSONWriter) Write(v interface{}) error { | ||
| if !w.enabled { | ||
| return nil | ||
| } | ||
|
|
||
| data, err := json.Marshal(v) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to marshal json output: %w", err) | ||
| } | ||
|
|
||
| if _, err := fmt.Fprintln(os.Stdout, string(data)); err != nil { | ||
| return fmt.Errorf("failed to write json output: %w", err) | ||
| } | ||
| return nil |
There was a problem hiding this comment.
JSONWriter.Write always writes to os.Stdout, which bypasses existing output routing (e.g., Report.out buffered writer) and makes it harder to test/redirect output. Consider accepting an io.Writer (or storing one on JSONWriter) so callers can write JSON to the same destination they use for other output.
| func (w *JSONWriter) Write(v interface{}) error { | ||
| if !w.enabled { | ||
| return nil | ||
| } | ||
|
|
||
| data, err := json.Marshal(v) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to marshal json output: %w", err) | ||
| } | ||
|
|
||
| if _, err := fmt.Fprintln(os.Stdout, string(data)); err != nil { | ||
| return fmt.Errorf("failed to write json output: %w", err) | ||
| } |
There was a problem hiding this comment.
Write marshals to a []byte, converts to string, then prints via fmt.Fprintln, adding extra allocations. Using json.NewEncoder(out).Encode(v) (with the chosen writer) avoids the string conversion and streams directly; it also naturally appends a newline.
| "io" | ||
| "os" | ||
| "strings" | ||
| "testing" | ||
| ) | ||
|
|
||
| func captureStdout(t *testing.T, fn func() error) (string, error) { | ||
| t.Helper() | ||
|
|
||
| originalStdout := os.Stdout | ||
| r, w, err := os.Pipe() | ||
| if err != nil { | ||
| t.Fatalf("failed to create pipe: %v", err) | ||
| } | ||
| os.Stdout = w | ||
|
|
||
| runErr := fn() | ||
|
|
||
| if err := w.Close(); err != nil { | ||
| t.Fatalf("failed to close writer: %v", err) | ||
| } | ||
| os.Stdout = originalStdout | ||
|
|
||
| var buf bytes.Buffer | ||
| if _, err := io.Copy(&buf, r); err != nil { | ||
| t.Fatalf("failed to read captured output: %v", err) | ||
| } | ||
| if err := r.Close(); err != nil { | ||
| t.Fatalf("failed to close reader: %v", err) | ||
| } |
There was a problem hiding this comment.
captureStdout mutates the global os.Stdout and doesn’t restore it if fn panics, which can cascade failures into other tests. Use defer/t.Cleanup to restore stdout and close pipe fds reliably (and consider guarding against parallel tests if any are added later).
| "io" | |
| "os" | |
| "strings" | |
| "testing" | |
| ) | |
| func captureStdout(t *testing.T, fn func() error) (string, error) { | |
| t.Helper() | |
| originalStdout := os.Stdout | |
| r, w, err := os.Pipe() | |
| if err != nil { | |
| t.Fatalf("failed to create pipe: %v", err) | |
| } | |
| os.Stdout = w | |
| runErr := fn() | |
| if err := w.Close(); err != nil { | |
| t.Fatalf("failed to close writer: %v", err) | |
| } | |
| os.Stdout = originalStdout | |
| var buf bytes.Buffer | |
| if _, err := io.Copy(&buf, r); err != nil { | |
| t.Fatalf("failed to read captured output: %v", err) | |
| } | |
| if err := r.Close(); err != nil { | |
| t.Fatalf("failed to close reader: %v", err) | |
| } | |
| "errors" | |
| "io" | |
| "os" | |
| "strings" | |
| "sync" | |
| "testing" | |
| ) | |
| var stdoutCaptureMu sync.Mutex | |
| func captureStdout(t *testing.T, fn func() error) (string, error) { | |
| t.Helper() | |
| stdoutCaptureMu.Lock() | |
| defer stdoutCaptureMu.Unlock() | |
| originalStdout := os.Stdout | |
| r, w, err := os.Pipe() | |
| if err != nil { | |
| t.Fatalf("failed to create pipe; verify the test environment supports os.Pipe and rerun the test: %v", err) | |
| } | |
| writerClosed := false | |
| readerClosed := false | |
| stdoutRestored := false | |
| defer func() { | |
| if !stdoutRestored { | |
| os.Stdout = originalStdout | |
| } | |
| if !writerClosed { | |
| if closeErr := w.Close(); closeErr != nil && !errors.Is(closeErr, os.ErrClosed) { | |
| t.Fatalf("failed to close stdout capture writer during cleanup; rerun the test after checking for pending writes: %v", closeErr) | |
| } | |
| } | |
| if !readerClosed { | |
| if closeErr := r.Close(); closeErr != nil && !errors.Is(closeErr, os.ErrClosed) { | |
| t.Fatalf("failed to close stdout capture reader during cleanup; rerun the test after checking for pending reads: %v", closeErr) | |
| } | |
| } | |
| }() | |
| os.Stdout = w | |
| runErr := fn() | |
| if err := w.Close(); err != nil { | |
| t.Fatalf("failed to close stdout capture writer; rerun the test after checking for pending writes: %v", err) | |
| } | |
| writerClosed = true | |
| os.Stdout = originalStdout | |
| stdoutRestored = true | |
| var buf bytes.Buffer | |
| if _, err := io.Copy(&buf, r); err != nil { | |
| t.Fatalf("failed to read captured output; rerun the test after checking that stdout writes completed: %v", err) | |
| } | |
| if err := r.Close(); err != nil { | |
| t.Fatalf("failed to close stdout capture reader; rerun the test after checking for pending reads: %v", err) | |
| } | |
| readerClosed = true |
| if testSetStatus == models.TestSetStatusFailed || testSetStatus == models.TestSetStatusPassed { | ||
|
|
||
| if !r.config.DisableANSI { | ||
| if testSetStatus == models.TestSetStatusFailed { | ||
| pp.SetColorScheme(models.GetFailingColorScheme()) | ||
| } else { | ||
| pp.SetColorScheme(models.GetPassingColorScheme()) | ||
| if r.config.JSONOutput { | ||
| if err := utils.NewJSONWriter(true).Write(testReport); err != nil { | ||
| utils.LogError(r.logger, err, "failed to print json testrun summary") | ||
| } | ||
| } else { | ||
|
|
||
| summaryFormat := "\n <=========================================> \n" + | ||
| " TESTRUN SUMMARY. For test-set: %s\n" + | ||
| "\tTotal tests: %s\n" + | ||
| "\tTotal test passed: %s\n" + | ||
| "\tTotal test failed: %s\n" | ||
| if !r.config.DisableANSI { |
There was a problem hiding this comment.
In JSON mode, keploy test prints one JSON object per test-set completion (multiple JSON documents separated by newlines). If the intent is a single valid JSON value for the whole command, consider aggregating and emitting one object/array at the end (or explicitly adopting/documenting NDJSON).
| if r.config.JSONOutput { | ||
| if err := utils.NewJSONWriter(true).Write(testReport); err != nil { | ||
| utils.LogError(r.logger, err, "failed to print json testrun summary") | ||
| } |
There was a problem hiding this comment.
The error log message failed to print json testrun summary doesn’t give a next step for the user. Consider including actionable guidance (e.g., that stdout may be closed/redirected, or suggesting rerunning without --json / checking output redirection).
|
Addressed bot findings in 0d3e099:
|
Add --json persistent flag to the root command. When set, keploy test and keploy report output structured JSON to stdout instead of formatted tables. Suppresses logo printing in JSON mode. Includes JSONWriter utility with tests. Relates to keploy#3883 This contribution was developed with AI assistance (Codex). Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
- [P1] Redirect logger to stderr in JSON mode to prevent log lines from contaminating structured JSON output on stdout - [P1] Recompute Success/Failure/Ignored/Obsolete counters after filtering by TestCaseIDs in JSON output (were stale before) - [P2] Fix config tag naming: json_output -> jsonOutput to match the camelCase convention used by all other config fields Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
f11f7f5 to
f3fce30
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
cli/provider/cmd.go:690
- In JSON mode you redirect logs to stderr, but subsequent logger reconfiguration paths in this function (e.g.,
ChangeLogLevelfor--debug,AddModeforagent/enable-testing, andChangeColorEncoding) rebuild the logger targeting stdout again. This can reintroduce log lines on stdout and break the contract of emitting clean JSON. To keep stdout clean, ensure all logger rebuilds respectJSONOutput(use stderr as the sink) or perform the stderr redirection as the final logger mutation step.
// In JSON mode, redirect logs to stderr so they don't contaminate JSON on stdout
if c.cfg.JSONOutput {
logger, err := log.RedirectToStderr()
if err == nil {
*c.logger = *logger
}
}
disableAnsi, _ := (cmd.Flags().GetBool("disable-ansi"))
// Skip printing logo for agent command to avoid duplicate logos in native mode
if cmd.Name() != "agent" && !c.cfg.JSONOutput {
PrintLogo(os.Stdout, disableAnsi)
}
if c.cfg.Debug {
logger, err := log.ChangeLogLevel(zap.DebugLevel)
*c.logger = *logger
if err != nil {
errMsg := "failed to change log level"
utils.LogError(c.logger, err, errMsg)
return errors.New(errMsg)
}
}
if c.cfg.Record.BasePath != "" {
port, err := pkg.ExtractPort(c.cfg.Record.BasePath)
if err != nil {
errMsg := "failed to extract port from base URL"
utils.LogError(c.logger, err, errMsg)
return errors.New(errMsg)
}
c.cfg.Port = port
c.cfg.E2E = true
}
// Add mode to logger for agent command to differentiate agent logs from client logs
if cmd.Name() == "agent" {
logger, err := log.AddMode(cmd.Name())
*c.logger = *logger
if err != nil {
errMsg := "failed to add mode to logger"
utils.LogError(c.logger, err, errMsg)
return errors.New(errMsg)
}
}
if c.cfg.EnableTesting {
// Add mode to logger to debug the keploy during testing
if cmd.Name() != "agent" { // Skip if already added for agent
logger, err := log.AddMode(cmd.Name())
*c.logger = *logger
if err != nil {
errMsg := "failed to add mode to logger"
utils.LogError(c.logger, err, errMsg)
return errors.New(errMsg)
}
}
c.cfg.DisableTele = true
}
if c.cfg.DisableANSI {
logger, err := log.ChangeColorEncoding()
models.IsAnsiDisabled = true
*c.logger = *logger
if err != nil {
errMsg := "failed to change color encoding"
utils.LogError(c.logger, err, errMsg)
return errors.New(errMsg)
}
c.logger.Info("Color encoding is disabled")
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // RedirectToStderr re-creates the logger writing to stderr instead of stdout. | ||
| // Use this when --json mode is active to prevent log lines from contaminating | ||
| // structured JSON on stdout. | ||
| func RedirectToStderr() (*zap.Logger, error) { | ||
| encoder := NewANSIConsoleEncoder(LogCfg.EncoderConfig) | ||
| core := zapcore.NewCore( | ||
| encoder, | ||
| zapcore.AddSync(os.Stderr), | ||
| LogCfg.Level, | ||
| ) | ||
|
|
||
| logger := zap.New(core) | ||
| return logger, nil | ||
| } |
| if r.config.JSONOutput { | ||
| if err := utils.NewJSONWriter(true).Write(testReport); err != nil { | ||
| utils.LogError(r.logger, err, "failed to print json testrun summary") | ||
| } | ||
| } else { |
| pp.SetColorScheme(models.GetPassingColorScheme()) | ||
| if r.config.JSONOutput { | ||
| if err := utils.NewJSONWriter(true).Write(testReport); err != nil { | ||
| utils.LogError(r.logger, err, "failed to print json testrun summary") |
| t.Fatalf("failed to create pipe: %v", err) | ||
| } | ||
| os.Stdout = w | ||
|
|
||
| runErr := fn() | ||
|
|
||
| if err := w.Close(); err != nil { | ||
| t.Fatalf("failed to close writer: %v", err) | ||
| } | ||
| os.Stdout = originalStdout | ||
|
|
||
| var buf bytes.Buffer | ||
| if _, err := io.Copy(&buf, r); err != nil { | ||
| t.Fatalf("failed to read captured output: %v", err) | ||
| } | ||
| if err := r.Close(); err != nil { | ||
| t.Fatalf("failed to close reader: %v", err) |
Summary
Adds
--jsonpersistent flag to the root command. When set,keploy testandkeploy reportoutput structured JSON to stdout instead of formatted tables.Demo
Why this matters
Modern CLI tools (gh, docker, kubectl) support JSON output for scripting. Keploy currently outputs only human-readable text with ANSI colors, making it hard to pipe into jq, feed CI dashboards, or analyze results programmatically.
Changes
--jsonpersistent flag on root cobra commandutils/output.gowithJSONWriterutility + testskeploy test: outputsTestReportas JSON when flag is setkeploy report: outputs structured JSON when flag is set--jsonis activeTesting
utils/output_test.gogo build ./...succeedsgo vetcleanRelates to #3883
This contribution was developed with AI assistance (Codex).