Skip to content

.NET: Request summaries with a trailing user message - #10219

Draft
Lulu (lulu0119) wants to merge 2 commits into
microsoft:mainfrom
lulu0119:codex/fix-summary-instruction-order
Draft

.NET: Request summaries with a trailing user message#10219
Lulu (lulu0119) wants to merge 2 commits into
microsoft:mainfrom
lulu0119:codex/fix-summary-instruction-order

Conversation

@lulu0119

@lulu0119 Lulu (lulu0119) commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Motivation & Context

With long coding histories, the summarizer can continue the old task instead of producing a summary.

Description & Review Guide

  • What are the major changes? Append one user summary request after the history, keep the leading system prompt, and update the existing ordering test. No public API changes.
  • What is the impact of these changes? Across three models and 12 coding histories (~100K tokens), summary-task success increased from 0/72 to 72/72 (+100 percentage points). The 72 baseline results were reused; 72 responses with the added message were newly generated. Vieval report · Full evidence.
  • What do you want reviewers to focus on? The final user message and provider compatibility.

Related Issue

Fixes #10221.

Draft pending maintainer agreement on the problem and proposed direction.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: implementation, test update, benchmark, and PR description.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
    • All 20 summarization tests pass. The full unit run has one unrelated Windows symlink-cleanup failure in FileAgentSkillLoaderTests.
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change.

Copilot AI balanced review requested due to automatic review settings November 10, 2026 08:17
@agent-framework-automation agent-framework-automation Bot added the .NET Usage: [Issues, PRs], Target: .Net label Nov 10, 2026
@lulu0119

Lulu (lulu0119) commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Full benchmark report and reproducibility bundle: 144 calls on real coding traces at approximately 100K input tokens. Includes the protocol, prompts, original outputs, grades, and analysis.

After extracting, run npm ci and npx vieval viewer reports/vieval. Viewing saved results requires no API key and makes no model calls.

SHA256: 7ed28d42031f64c97022ea85c806887422c8981cec1e8ac2072ccb7860b512fc

summary-order-coding-100k.zip

Submission containing materials of a third party: The attached bundle includes selected Nebius SWE-rebench OpenHands trajectories (Trofimova et al., 2025), obtained from the ThoughtWorks normalized distribution at revision cef72d1f4d0caabf85937adf8337a14b7522c782. The selected Nebius trajectory data is licensed under CC BY 4.0. Preparation selects continuous prefixes, removes original agent system messages, and decodes stored tool calls. Dataset cards, source identifiers, attribution, and preparation details are retained in the bundle. Repository code excerpts remain subject to their upstream licenses.

Copilot AI reviewed Nov 10, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The mock-only ordering test leaves external provider acceptance and instruction semantics of a trailing system message unverified, requiring maintainer validation.

No open issues 🎉

What changed in this PR

This PR moves the summarization instruction after the selected conversation history in .NET’s compaction strategy, aiming to keep the model focused on summarizing rather than continuing the previous task.

Changes:

  • Append the unchanged system summary prompt after the selected history.
  • Update the existing unit test to assert the new message order.
  • Preserve public APIs and prompt text.
File Description
dotnet/​tests/​Microsoft.Agents.AI.UnitTests/​Compaction/​SummarizationCompactionStrategyTests.cs Updates assertions to expect history before the custom system prompt.
dotnet/​src/​Microsoft.Agents.AI/​Compaction/​SummarizationCompactionStrategy.cs Appends the summarization system instruction after collecting history.

Review effort: Balanced

@lulu0119

Lulu (lulu0119) commented Nov 10, 2026

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@lulu0119 Lulu (lulu0119) mentioned this pull request Nov 10, 2026
Open
2 of 3 tasks
@lulu0119

Lulu (lulu0119) commented Nov 10, 2026

Copy link
Copy Markdown
Author

I overlooked the issue-first requirement in CONTRIBUTING.md and opened this draft PR before obtaining maintainer agreement. Sorry for the process mistake. I have now opened #10221 and will keep this PR in draft pending confirmation of the problem and proposed direction.

@lulu0119 Lulu (lulu0119) changed the title .NET: Append summarization instructions after history .NET: Request summaries with a trailing user message Nov 10, 2026

This branch was previously deployed

1 inactive deployment
github-app-auth — ab87dad4 Deployed Nov 10, 2026 by lulu0119 via add_label #27469
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@eavanvalkenburg Eduard van Valkenburg (eavanvalkenburg) Awaiting requested review from eavanvalkenburg eavanvalkenburg will be requested when the pull request is marked ready for review eavanvalkenburg is a code owner
@SergeyMenshykh SergeyMenshykh Awaiting requested review from SergeyMenshykh SergeyMenshykh will be requested when the pull request is marked ready for review SergeyMenshykh is a code owner
@rogerbarreto Roger Barreto (rogerbarreto) Awaiting requested review from rogerbarreto rogerbarreto will be requested when the pull request is marked ready for review rogerbarreto is a code owner
@westey-m westey (westey-m) Awaiting requested review from westey-m westey-m will be requested when the pull request is marked ready for review westey-m is a code owner
@baywet Vincent Biret (baywet) Awaiting requested review from baywet baywet will be requested when the pull request is marked ready for review baywet is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

.NET Usage: [Issues, PRs], Target: .Net

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

.NET: [Bug]: Summarization can continue historical coding tasks with a leading summary instruction

2 participants

@lulu0119