You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature]: Add pre-execution validation for agent actions #1509
AgentScope is an open-source project. To involve a broader community, we recommend asking your questions in English.
Is your feature request related to a problem? Please describe.
AgentScope enables multi-agent systems where agents coordinate tasks and trigger actions such as tool calls and API requests.
In these systems, execution depends on the final payload produced at runtime, which may evolve across multiple steps or agent interactions.
As workflows become more complex, it can be difficult to ensure that the final action being executed is complete, expected, and consistent with system constraints.
Describe the solution you'd like
Introduce a validation step before execution that verifies:
agent identity and context
action being performed
parameters (schema + expected values)
destination or external system
timestamp / validity window
optional nonce or replay protection
If validation fails → execution does not occur.
This could be implemented as a wrapper around execution functions or as a system-level enforcement layer to ensure consistent behavior across workflows.
Describe alternatives you've considered
Validation can be implemented within individual agents or workflow logic, but this makes enforcement optional and dependent on how each system is configured.
A system-level validation layer would ensure consistent behavior across all execution paths without requiring each agent or workflow to implement validation independently.
Additional context
As agent systems move toward production use cases, ensuring correctness at the execution boundary helps improve reliability and predictability across complex workflows.
This issue is marked as stale because there has been no activity for 60 days. Remove stale label or add new comments or this issue will be closed in 90 day.
#1509 proposes pre-execution validation for agent actions with a concrete checklist. We built exactly that layer, ran it through the AgentDojo official benchmark with a real LLM in the loop, and are publishing the artifacts so the community can audit every number. This issue is a working implementation path for #1509, not a competing proposal.
Closed-loop results (each reported number is backed by a checksummed artifact in the repo)
AgentDojo official benchmark, real LLM in the loop (97 tasks × 2 rounds — the second round is a repeat run for stability, not independent tasks; official injection suite, official security() judgment): no ASR/FP events observed in the v2.2 re-run (zero-event result; sample size limits confidence)
Published fix-cycle trajectory FP 16 → 1 → 0 across v1.2 → v2.1 → v2.2 (same suite and judgment caliber across versions) — same-source iteration, failures included
Single-model (GLM) spot check (24 benign + 24 attack): FP = 0, ASR = 0 — not a cross-model claim
Synthetic stress layer (100,000 self-constructed seeded scenarios, 7 rule families): FN = 0, FP = 0; interception Wilson-95 lower bound 99.99%+ on this synthetic distribution — not extrapolable to real attack distributions
White-box adaptive attacks: 49 cases across 16 vector families — 43 hard-blocked, 6 edge cases documented in detail (none a full bypass; per-case disposition in the artifacts)
Design properties relevant to AgentScope
Judge is fixed-constant pure math: no LLM call in the enforcement path, no learned parameters → microsecond-scale per-action cost (P99 = 0.13 ms stress suite) and reproducible verdicts
Verifier verdicts are deterministic and seeded — same inputs always yield the same verdict (the LLM itself is not deterministic), so the judgment layer can sit inside an auditable pipeline
The layer complements monitoring and alignment training; it does not replace them
Honest boundaries
Synthetic scenarios are abstracted from publicly disclosed incident categories, not production traffic. Same-source fix cycles are not independent stability trials. GLM subset vs full run differ in model and sample size and are not directly comparable.
Verify it yourself python verify_artifacts.py recomputes every SHA-256 chain and the Wilson-95 bounds from raw counts — stdlib only, no trust required.
Feedback on the methodology is very welcome. If the maintainers see a fit, we'd align the contract with AgentScope's execution seams (and with the middleware direction in #1712 / #2022 where relevant).
I'd like to take this on as a focused, runnable example using the existing ToolMiddlewareBase.on_tool_call execution boundary.
For a first PR, I propose an offline notification tool with a host-provided execution context and policy. The innermost middleware will validate the final argument schema, allowed action/recipient, and context expiry before calling next_handler; rejected calls will return a denied tool result without invoking the function. Tests will check the actual side-effect record, including malformed input and an outer middleware rewriting a previously allowed recipient.
This keeps the change within examples/tests and a companion English/Chinese docs update, with no new dependencies or public APIs. It demonstrates a pre-execution gate for explicitly wrapped tools, not global enforcement, authentication, or durable replay protection. I will use Refs #1509 so the broader system-level contract remains open for discussion.
Please assign this focused contribution to me if available.
It's yours, @yang0228. If there is no pull request and no word from you by 2026-11-20 the claim is released so nobody is blocked — a comment here renews it.
Opened #3793 with the focused offline pre-execution validation example and regression tests. Companion English/Chinese docs: agentscope-ai/docs#99.
The guard runs last in the tool middleware chain and checks the final schema, action/recipient, and trusted host context/expiry before invoking the function. Tests verify that denied calls produce no function side effects, including arguments rewritten by an outer middleware and the final Toolkit DENIED result.
Validation: all-file pre-commit and normal commit hooks passed; 43 related tests and 12 subtests passed; the offline demo records exactly one notification. The full suite reported 3336 passed, 125 skipped, and 1821 subtests passed, with 22 existing Docker setup errors because the host uv version is below the fixture's minimum. The PR documents this local verification limit.
This is intentionally an example-level first step using existing APIs (Refs #1509), not global enforcement, identity authentication, or durable replay protection. The broader execution contract remains open for discussion.
I am extending PR #3793 from the initial example into a reusable tool-level validation helper, following the wrapper approach described in this issue.
The proposed ToolValidationMiddleware accepts a synchronous application validator, validates the final business-argument schema, and returns DENIED without dispatch when the validator supplies a rejection reason. Only None permits execution; invalid validator results and programming errors fail closed. Application identity/context, allowed actions and destinations, and validity checks stay in a trusted host-bound callback rather than in model-supplied authority fields.
The helper will isolate business-argument snapshots, reject validator-side rewriting, and preserve AgentScope's internal state injection and permission behavior. It remains the innermost middleware on selected tools, not a global security sandbox. The existing example will demonstrate the full checklist using this reusable API; optional nonce/replay policy remains application-specific.
I will also address the timing suggestions on the PR with deterministic tests for exact expiry, deferred invocation, and attempted argument changes around validation/dispatch. The tests and documentation will distinguish admission-time validation from revoking an already-started action.
Updated #3793 to implement the wrapper option requested here with reusable agentscope.tool.ToolValidationMiddleware.
The library helper checks final business-argument schemas and a synchronous host-bound policy, denies rejected actions before dispatch, isolates nested input snapshots, rejects validator rewrites, and preserves existing state injection and permission behavior. The example demonstrates identity/workflow context, action, expected parameter values, destination and expiry using the public API.
The PR also addresses the timing feedback with exact-expiry, deferred-consumption and deterministic scheduling regressions. Expiry is an admission check; strict deadlines, global enforcement and optional durable replay protection are not claimed.
Validation is now fully green locally: 3372 tests and 1833 subtests passed, 125 skipped, with no failures or errors. The earlier Docker setup limitation was resolved by using an isolated uv 0.13.7. All-files pre-commit, normal commit hooks and independent review passed.
The PR now has a closing association (Fixes #1509). Companion EN/ZH documentation has been updated in agentscope-ai/docs#99 and should merge after the source PR.
Activity
github-actions commentedon Jun 20, 2026
This issue is marked as stale because there has been no activity for 60 days. Remove stale label or add new comments or this issue will be closed in 90 day.
PHM-Audit commentedon Oct 13, 2026
Title:
Body:
#1509 proposes pre-execution validation for agent actions with a concrete checklist. We built exactly that layer, ran it through the AgentDojo official benchmark with a real LLM in the loop, and are publishing the artifacts so the community can audit every number. This issue is a working implementation path for #1509, not a competing proposal.
Point-by-point against the #1509 checklist
PASS/VETOverdict at the dispatch point;VETOblocks before side effectsverifier_interface.pyiRepo: https://github.com/PHM-Audit/agent-action-verifier
Closed-loop results (each reported number is backed by a checksummed artifact in the repo)
security()judgment): no ASR/FP events observed in the v2.2 re-run (zero-event result; sample size limits confidence)Design properties relevant to AgentScope
Honest boundaries
Synthetic scenarios are abstracted from publicly disclosed incident categories, not production traffic. Same-source fix cycles are not independent stability trials. GLM subset vs full run differ in model and sample size and are not directly comparable.
Verify it yourself
python verify_artifacts.pyrecomputes every SHA-256 chain and the Wilson-95 bounds from raw counts — stdlib only, no trust required.Feedback on the methodology is very welcome. If the maintainers see a fit, we'd align the contract with AgentScope's execution seams (and with the middleware direction in #1712 / #2022 where relevant).
yang0228 commentedon Nov 6, 2026
I'd like to take this on as a focused, runnable example using the existing
ToolMiddlewareBase.on_tool_callexecution boundary.For a first PR, I propose an offline notification tool with a host-provided execution context and policy. The innermost middleware will validate the final argument schema, allowed action/recipient, and context expiry before calling
next_handler; rejected calls will return a denied tool result without invoking the function. Tests will check the actual side-effect record, including malformed input and an outer middleware rewriting a previously allowed recipient.This keeps the change within examples/tests and a companion English/Chinese docs update, with no new dependencies or public APIs. It demonstrates a pre-execution gate for explicitly wrapped tools, not global enforcement, authentication, or durable replay protection. I will use
Refs #1509so the broader system-level contract remains open for discussion.Please assign this focused contribution to me if available.
github-actions commentedon Nov 6, 2026
It's yours, @yang0228. If there is no pull request and no word from you by 2026-11-20 the claim is released so nobody is blocked — a comment here renews it.
yang0228 commentedon Nov 6, 2026
Opened #3793 with the focused offline pre-execution validation example and regression tests. Companion English/Chinese docs: agentscope-ai/docs#99.
The guard runs last in the tool middleware chain and checks the final schema, action/recipient, and trusted host context/expiry before invoking the function. Tests verify that denied calls produce no function side effects, including arguments rewritten by an outer middleware and the final Toolkit
DENIEDresult.Validation: all-file pre-commit and normal commit hooks passed; 43 related tests and 12 subtests passed; the offline demo records exactly one notification. The full suite reported 3336 passed, 125 skipped, and 1821 subtests passed, with 22 existing Docker setup errors because the host uv version is below the fixture's minimum. The PR documents this local verification limit.
This is intentionally an example-level first step using existing APIs (
Refs #1509), not global enforcement, identity authentication, or durable replay protection. The broader execution contract remains open for discussion.yang0228 commentedon Nov 6, 2026
I am extending PR #3793 from the initial example into a reusable tool-level validation helper, following the wrapper approach described in this issue.
The proposed
ToolValidationMiddlewareaccepts a synchronous application validator, validates the final business-argument schema, and returnsDENIEDwithout dispatch when the validator supplies a rejection reason. OnlyNonepermits execution; invalid validator results and programming errors fail closed. Application identity/context, allowed actions and destinations, and validity checks stay in a trusted host-bound callback rather than in model-supplied authority fields.The helper will isolate business-argument snapshots, reject validator-side rewriting, and preserve AgentScope's internal state injection and permission behavior. It remains the innermost middleware on selected tools, not a global security sandbox. The existing example will demonstrate the full checklist using this reusable API; optional nonce/replay policy remains application-specific.
I will also address the timing suggestions on the PR with deterministic tests for exact expiry, deferred invocation, and attempted argument changes around validation/dispatch. The tests and documentation will distinguish admission-time validation from revoking an already-started action.
yang0228 commentedon Nov 6, 2026
Updated #3793 to implement the wrapper option requested here with reusable
agentscope.tool.ToolValidationMiddleware.The library helper checks final business-argument schemas and a synchronous host-bound policy, denies rejected actions before dispatch, isolates nested input snapshots, rejects validator rewrites, and preserves existing state injection and permission behavior. The example demonstrates identity/workflow context, action, expected parameter values, destination and expiry using the public API.
The PR also addresses the timing feedback with exact-expiry, deferred-consumption and deterministic scheduling regressions. Expiry is an admission check; strict deadlines, global enforcement and optional durable replay protection are not claimed.
Validation is now fully green locally: 3372 tests and 1833 subtests passed, 125 skipped, with no failures or errors. The earlier Docker setup limitation was resolved by using an isolated uv 0.13.7. All-files pre-commit, normal commit hooks and independent review passed.
The PR now has a closing association (
Fixes #1509). Companion EN/ZH documentation has been updated in agentscope-ai/docs#99 and should merge after the source PR.