Degrade unreachable non-critical MCP servers instead of aborting gateway startup by Copilot · Pull Request #52075 · github/gh-aw

Copilot · 2026-08-11T14:36:47Z

A single unreachable MCP server (Datadog returning HTTP 503) aborted the entire gateway startup on Smoke OTEL, taking Sentry, Grafana, and OTLP export down with it even though those three servers connected successfully.

check_mcp_servers.sh already had degradation logic keyed off required: false, but it was unreachable end to end: the frontmatter schema rejected required at compile time, and the flag would not have survived into gateway-output.json — the file the check script actually reads — because the gateway configuration spec has no such field.

mcp-servers:
  datadog:
    type: http
    url: "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=core"
    required: false   # startup failure logs a warning; other servers keep running

Schema and validation

main_workflow_schema.json / mcp_config_schema.json: required (boolean, default true) is now valid on stdio and HTTP MCP server definitions.
mcp_config_validation.go, mcp_property_validation.go: added required to the known-field maps. The renderer already knew how to emit "required": false; only validation was blocking it.

Flag propagation

start_mcp_gateway.cjs: new extractOptionalServerNames() collects servers marked non-critical, strips the flag from the config handed to the gateway (unknown field per spec), logs which servers are best-effort, and forwards the names to the check script via GH_AW_MCP_OPTIONAL_SERVERS.
check_mcp_servers.sh: honors that env var in addition to .required == false, and emits a distinct line — ⚠ <name>: non-critical MCP server unavailable, continuing without it — instead of the generic optional-failure message.

Workflow and docs

shared/mcp/datadog.md: Datadog marked required: false; smoke-otel-backends.lock.yml and mcp-inspector.lock.yml recompiled.
New "Required Field" section in reference/tools.md.

Servers without the flag remain startup-critical, and startup still fails when no server connects at all.

Tests

Go: "required": false renders only for the non-critical server.
vitest: extractOptionalServerNames collection and flag stripping.
shell: env-var-driven optional path, actionable log line, and that servers absent from the list stay fatal.

Not addressed here: the issue's suggested circuit breaker for repeated transient 503s within a 6h window. Optional servers still run the full 10s/20s/30s retry sequence — they just no longer abort startup.

Fixes [aw-failures] [P0] MCP gateway hard-fails on one unreachable server (Datadog 503) — kills Smoke OTEL #52062

Run: https://github.com/github/gh-aw/actions/runs/31508439729> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 35.1 AIC · ⌖ 6.14 AIC · ⊞ 8.5K · ◷

_{Comment /souschef to run again}

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 12.5 AIC · ⌖ 3.4 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 7.47 AIC · ⌖ 11.6 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 11.1 AIC · ⌖ 3.4 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Requested branch update from https://github.com/github/gh-aw/actions/runs/31535473039.> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 8.59 AIC · ⌖ 5.18 AIC · ⊞ 8.5K · ◷

_{Comment /souschef to run again}

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 14.9 AIC · ⌖ 5.12 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

…ay startup Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…eway handoff Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

Adds per-server startup criticality so unavailable non-critical MCP servers no longer abort the gateway.

Changes:

Adds and propagates the required MCP setting.
Degrades optional-server failures while retaining fatal defaults.
Updates Datadog workflows, documentation, and tests.

Show a summary per file

File	Description
`pkg/workflow/mcp_property_validation.go`	Allows `required` during schema projection.
`pkg/workflow/mcp_config_validation.go`	Recognizes the new field.
`pkg/workflow/mcp_config_compilation_test.go`	Tests rendered criticality configuration.
`pkg/parser/schemas/mcp_config_schema.json`	Defines `required`.
`pkg/parser/schemas/main_workflow_schema.json`	Enables the field for MCP servers.
`docs/src/content/docs/reference/tools.md`	Documents startup criticality.
`actions/setup/sh/check_mcp_servers.sh`	Degrades optional-server failures.
`actions/setup/sh/check_mcp_servers_test.sh`	Tests shell degradation behavior.
`actions/setup/js/start_mcp_gateway.test.cjs`	Tests extraction and stripping.
`actions/setup/js/start_mcp_gateway.cjs`	Propagates optional server names.
`.github/workflows/smoke-otel-backends.lock.yml`	Recompiles Smoke OTEL configuration.
`.github/workflows/shared/mcp/datadog.md`	Marks Datadog non-critical.
`.github/workflows/mcp-inspector.lock.yml`	Recompiles inspector configuration.

Review details

Tip

Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Files reviewed: 13/13 changed files
Comments generated: 1
Review effort level: Balanced

Copilot · 2026-08-11T19:43:53Z

+	tmpFile.Close()
+
+	compiler := NewCompiler()
+	compiler.SetSkipValidation(true)


Updated in 2ad15f7: removed compiler.SetSkipValidation(true) from TestMCPServerRequiredFalseCompilation, so the test now exercises frontmatter/schema validation as well as YAML rendering.

gh-aw-bot · 2026-08-11T15:50:50Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

No failed checks were reported in the compact candidate set for this run.
Branch update was requested by sous-chef in run https://github.com/github/gh-aw/actions/runs/31508439729.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 35.1 AIC · ⌖ 6.14 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Copilot · 2026-08-11T16:05:06Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

No failed checks were reported in the compact candidate set for this run....

Done. Branch is refreshed at 080914a (includes latest main), and I completed a PR-finisher verification pass. Local checks passed: make fmt, make lint, make test-unit, make test, and make recompile. I also reviewed recent workflow-run/job logs for this branch; no failed jobs were reported for the latest run I inspected (31509255911).

github-actions · 2026-08-11T18:37:13Z

PR Triage: #52075

Category: bug
Risk: medium
Priority score: 75/100 (impact 35/50, urgency 25/30, quality 15/20) → high
Recommended action: fast_track

Automated triage — see labels for machine-readable classification.

Generated by 🔧 PR Triage Agent · auto · 45 AIC · ⌖ 2.09 AIC · ⊞ 7.8K · ◷

gh-aw-bot · 2026-08-11T18:47:00Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

No failed checks were reported in the compact candidate set for this run.
Branch update was requested by sous-chef in run https://github.com/github/gh-aw/actions/runs/31523880210.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 12.5 AIC · ⌖ 3.4 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Copilot · 2026-08-11T19:01:51Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

No failed checks were reported in the compact candidate set for this run....

Done. Branch is refreshed at 5e7ff60 (up to date with main 3b4ef7e), and I completed a pr-finisher verification pass. Local checks passed: make fmt, make lint, make test-unit, make test, and make recompile. I also reviewed branch workflow runs and checked failed-job logs for 31524515041; no failed jobs were reported.

gh-aw-bot · 2026-08-11T19:22:38Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

Unresolved review threads to address (newest first):

copilot-pull-request-reviewer: Degrade unreachable non-critical MCP servers instead of aborting gateway startup #52075 (comment)

No failed checks were reported in the compact candidate set for this run.

Branch update was requested by sous-chef in run https://github.com/github/gh-aw/actions/runs/31527012599.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 7.47 AIC · ⌖ 11.6 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-08-11T19:44:53Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

Unresolved review threads to address (newest first):...

Done. Branch is refreshed at 2ad15f7 and I addressed the unresolved review feedback in that commit by enabling validation in TestMCPServerRequiredFalseCompilation (removed SetSkipValidation(true)). I verified locally with targeted and full checks: go test ./pkg/workflow -run TestMCPServerRequiredFalseCompilation, make fmt, make lint, make test-unit, make test, and make recompile (plus BASE_REF=origin/main make agent-report-progress). I also reviewed branch CI snapshot and failed-job logs for the latest completed CJS run before this push (31527547423): no failed jobs were reported.

gh-aw-bot · 2026-08-11T20:14:40Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

Unresolved review threads to address (newest first):

copilot-pull-request-reviewer: Degrade unreachable non-critical MCP servers instead of aborting gateway startup #52075 (comment)

No failed checks were reported in the compact candidate set for this run.

Branch update was requested by sous-chef in run https://github.com/github/gh-aw/actions/runs/31531364207.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 11.1 AIC · ⌖ 3.4 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Copilot · 2026-08-11T20:31:06Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

Unresolved review threads to address (newest first):...

Branch is already refreshed at 5d77ac2 (includes latest main merge cb03b8c). I re-verified locally with make fmt, make lint, make test-unit, make test, and make recompile — all passed. I also checked branch workflow runs and pulled failed-job logs for run 31531933076; there were no failed jobs. The unresolved thread at #discussion_r3759316776 is already addressed by commit 2ad15f7 and the existing thread reply.

gh-aw-bot · 2026-08-11T21:04:06Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

No failed checks were reported in the compact candidate set for this run.

Branch update was requested by sous-chef in run https://github.com/github/gh-aw/actions/runs/31535473039.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 8.59 AIC · ⌖ 5.18 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Copilot · 2026-08-11T21:29:18Z

@copilot Please refresh this branch, verify checks, and use the pr-finisher skill before handing back.

No failed checks were reported in the compact candidate set for this run....

Branch is refreshed at 4d55c04 (merged latest main c993af6). I re-verified with make fmt, make lint, make test-unit, make test, and make recompile; all passed locally. I also checked recent branch workflow runs and pulled failed-job logs for 31536114367, 31536114543, 31536114583, and 31536114528; no failed jobs were reported.

gh-aw-bot · 2026-08-11T22:04:09Z

@copilot This PR looks ready for another finishing pass. Please refresh the branch context, address any remaining reviewer feedback, and run the pr-finisher skill before handing back to maintainers.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 14.9 AIC · ⌖ 5.12 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Copilot · 2026-08-11T22:18:04Z

@copilot This PR looks ready for another finishing pass. Please refresh the branch context, address any remaining reviewer feedback, and run the pr-finisher skill before handing back to maintainers....

Branch context is refreshed and up to date at 4d55c04 (includes current origin/main c993af6). I ran a fresh finishing pass: make fmt, make lint, make test-unit, make test, and make recompile all passed locally. I also checked recent branch workflow runs and fetched failed-job logs for 31536114367, 31536114528, 31536114583, and 31536114543; no failed jobs were reported. The remaining unresolved review thread is already substantively addressed in 2ad15f7 and prior thread reply.

Initial plan

7d4336d

Copilot AI assigned Copilot and pelikhan Aug 11, 2026

Copilot started work on behalf of pelikhan August 11, 2026 14:37 View session

Copilot AI linked an issue Aug 11, 2026 that may be closed by this pull request

[aw-failures] [P0] MCP gateway hard-fails on one unreachable server (Datadog 503) — kills Smoke OTEL #52062

Closed

Copilot AI and others added 2 commits August 11, 2026 15:02

Degrade unreachable non-critical MCP servers instead of failing gatew…

2fe2025

…ay startup Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Clarify that the required flag is stripped for all servers before gat…

2601bc8

…eway handoff Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix MCP gateway hard-fail on unreachable server~~ Degrade unreachable non-critical MCP servers instead of aborting gateway startup Aug 11, 2026

Copilot finished work on behalf of pelikhan August 11, 2026 15:12

Copilot AI requested a review from pelikhan August 11, 2026 15:12

pelikhan marked this pull request as ready for review August 11, 2026 15:22

Copilot AI balanced review requested due to automatic review settings August 11, 2026 15:22

Copilot started reviewing on behalf of pelikhan August 11, 2026 15:23 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

Merge branch 'main' into copilot/aw-failures-degrade-unreachable-server

080914a

github-actions Bot mentioned this pull request Aug 11, 2026

[pr-sous-chef] PR Sous Chef run report - 2026-08-11T01:46Z #51906

Open

Copilot started work on behalf of gh-aw-bot August 11, 2026 15:51 View session

Copilot finished work on behalf of gh-aw-bot August 11, 2026 16:05

Copilot AI requested a review from gh-aw-bot August 11, 2026 16:05

github-actions Bot added otel pr-action:fast_track pr-agent:copilot pr-priority:high pr-risk:medium pr-type:bug labels Aug 11, 2026

github-actions Bot mentioned this pull request Aug 11, 2026

[PR Triage Report] PR Triage Report — 2026-08-11 (Run 31522790297) #52130

Open

Merge branch 'main' into copilot/aw-failures-degrade-unreachable-server

5e7ff60

Copilot started work on behalf of gh-aw-bot August 11, 2026 18:47 View session

Copilot finished work on behalf of gh-aw-bot August 11, 2026 19:03

Merge branch 'main' into copilot/aw-failures-degrade-unreachable-server

de3d17b

Copilot started work on behalf of gh-aw-bot August 11, 2026 19:23 View session

test: enable validation in MCP required compilation test

2ad15f7

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot August 11, 2026 19:45

Merge branch 'main' into copilot/aw-failures-degrade-unreachable-server

5d77ac2

Copilot started work on behalf of gh-aw-bot August 11, 2026 20:15 View session

Copilot finished work on behalf of gh-aw-bot August 11, 2026 20:31

Merge branch 'main' into copilot/aw-failures-degrade-unreachable-server

4d55c04

Copilot started work on behalf of gh-aw-bot August 11, 2026 21:04 View session

Copilot finished work on behalf of gh-aw-bot August 11, 2026 21:29

Copilot started work on behalf of gh-aw-bot August 11, 2026 22:04 View session

Copilot finished work on behalf of gh-aw-bot August 11, 2026 22:18

pelikhan merged commit 5dbb41a into main Aug 11, 2026
40 checks passed

pelikhan deleted the copilot/aw-failures-degrade-unreachable-server branch August 11, 2026 22:45

Uh oh!

Conversation

Copilot AI commented Aug 11, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Schema and validation

Flag propagation

Workflow and docs

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Copilot AI Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

gh-aw-bot commented Aug 11, 2026

Uh oh!

Copilot AI commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

PR Triage: #52075

Uh oh!

gh-aw-bot commented Aug 11, 2026

Uh oh!

Copilot AI commented Aug 11, 2026

Uh oh!

gh-aw-bot commented Aug 11, 2026

Uh oh!

Copilot AI commented Aug 11, 2026

Uh oh!

gh-aw-bot commented Aug 11, 2026

Uh oh!

Copilot AI commented Aug 11, 2026

Uh oh!

gh-aw-bot commented Aug 11, 2026

Uh oh!

Copilot AI commented Aug 11, 2026

Uh oh!

gh-aw-bot commented Aug 11, 2026

Uh oh!

Copilot AI commented Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Aug 11, 2026 •

edited by github-actions Bot

Loading