fix: bound YAML anchor/alias expansion to prevent OOM (billion laughs) - #3000

Problem

The YAML reader is exposed to an uncontrolled-resource-consumption ("billion laughs") denial of service (CWE-400). A tiny YAML document (well under 1 KB) using nested anchors/aliases can force the process to allocate many gigabytes and be OOM-killed. See https://portal.microsofticm.com/imp/v5/incidents/details/31000000667626/summary

Root cause

OpenApiYamlReader parses YAML via SharpYaml's YamlStream.Load, which produces a DAG where an alias (*a) resolves to the same shared YamlNode instance — so the parsed graph stays small. The explosion happens in YamlConverter.ToJsonNode, which converts that DAG into a System.Text.Json JsonNode tree. Because JsonNode is single-parent (a node cannot be attached to two parents), every alias occurrence must be materialized as an independent copy. With no bound, N nested anchors each referenced k times expand to k^N nodes → exponential memory → OOM.

Deduplication/sharing is not possible (single-parent constraint), so the only viable fix is to bound the work and fail fast.

Fix

Add a conversion budget threaded through YamlConverter.ToJsonNode that enforces two limits:

Rationale for the values:

• Depth 64 mirrors the default System.Text.Json MaxDepth already enforced on the JSON reader path (JsonNode.Parse), so this brings YAML to parity — any document deeper than 64 already fails today when supplied as JSON.
• Node count 5,000,000 is comfortably above legitimate specs (whose node count is linear in document size when they don't rely on alias fan-out), while a bomb trips almost immediately because the counter measures the expansion.

On breach the budget throws OpenApiReaderException, which OpenApiYamlReader.Read converts into an OpenApiDiagnostic error (Document = null) — consistent with the existing JsonException handling — instead of allowing an OOM.

Configurable limits

The two limits are exposed as public static uint properties on YamlConverter so consumers are never blocked by the defaults:

• YamlConverter.MaxDepth (default YamlConverter.DefaultMaxDepth = 64)
• YamlConverter.MaxNodeCount (default YamlConverter.DefaultMaxNodeCount = 5,000,000)

A consumer that must ingest an unusually large/deep-but-trusted document can raise the limits; a consumer that only ever parses small documents can lower them to fail faster. uint makes the non-negative intent explicit at the type level (negative literals fail to compile), and the setters reject 0. These are the only additions to the public API (recorded in PublicAPI.Unshipped.txt).

Note the limits are process-wide static state, best configured once at startup. They are an escape hatch for the defaults, not per-parse/per-thread configuration.

Tests

• YamlConverterTests.ExponentialAliasExpansionIsRejected — a nested anchor/alias bomb is rejected instead of exhausting memory.
• YamlConverterTests.ExcessiveNestingDepthIsRejected — nesting beyond the depth limit is rejected.
• YamlConverterTests.LegitimateAliasesStillConvert — normal alias usage still converts correctly.
• YamlConverterTests.ConversionLimitsDefaultToDocumentedValues — the properties expose the documented defaults.
• YamlConverterTests.SettingMaxDepthToZeroThrows / SettingMaxNodeCountToZeroThrows — zero limits are rejected and leave the effective limit unchanged.
• YamlConverterTests.RaisingMaxDepthAllowsDocumentsDeeperThanTheDefault — raising the limit permits a document deeper than the default.
• OpenApiYamlReaderTests.ReadReturnsDiagnosticErrorForExponentialAliasExpansion — the reader surfaces a diagnostic error (no document), not a throw/OOM.

Full Microsoft.OpenApi.Readers.Tests suite passes.

Validation on a large real-world spec (no false positives)

To confirm the limits do not reject legitimately large production descriptions, the Microsoft Graph beta OpenAPI document was loaded end-to-end through the patched reader.

Result under the patched build (with default limits):

The largest realistic production spec sits well below both caps, while the exponential bomb (theoretical ~387M nodes) is rejected almost immediately. The limits target the exponential pathology, not document size.

Notes / scope

• This is distinct from CVE-2026-49451 (circular $ref stack overflow), which was already fixed in 3.5.4.
• ReadFragment is still protected by the budget (it throws rather than OOM) but does not convert the exception to a diagnostic — left out of scope intentionally; happy to extend if preferred.
• The same fix applies byte-for-byte to support/v2; a companion PR will follow. support/v1 uses a different YAML parsing path and needs a separate assessment.