Actions: Add experimental queries for AI output validation CWE 1426#21678
Draft
data-douser wants to merge 2 commits intogithub:mainfrom
Draft
Actions: Add experimental queries for AI output validation CWE 1426#21678data-douser wants to merge 2 commits intogithub:mainfrom
data-douser wants to merge 2 commits intogithub:mainfrom
Conversation
Add detection for improper validation of AI-generated output (CWE-1426) in GitHub Actions workflows where AI action output flows unsanitized to code execution sinks. New query: - ImproperValidationOfAiOutputCritical.ql: Detects AI-generated output flowing to run steps or subsequent AI prompts in privileged contexts (severity 9.0) New library: - ImproperValidationOfAiOutputQuery.qll: Taint tracking from AI action output references to code execution and AI inference sinks MaD model (ai_inference_actions.model.yml): - 15 AI actions identified as AI inference sources whose outputs should be treated as untrusted
| @@ -0,0 +1 @@ | |||
| experimental/Security/CWE-1426/ImproperValidationOfAiOutputCritical.ql | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Actions: Add experimental CWE-1426 improper AI output validation query
Description
Adds detection for improper validation of AI-generated output (CWE-1426) in GitHub Actions workflows where AI action output flows unsanitized into code execution sinks (run steps, subsequent AI prompts).
This is the complementary query to CWE-1427 (prompt injection). While CWE-1427 detects user-controlled data flowing into AI prompts, CWE-1426 detects AI-generated output flowing out of AI actions into executable contexts — enabling chained injection attacks where a successful prompt injection produces malicious AI output that executes as code in subsequent workflow steps.
New query
actions/improper-ai-output-handling/criticalrun:steps, MaD-defined code-injection sinks, or subsequent AI prompts in privileged contextsNew library and MaD models
ImproperValidationOfAiOutputQuery.qll: DefinesAiInferenceOutputSource(identifies AI action step outputs via MaDai-inferencekind),ImproperAiOutputSink(run steps, code-injection sinks, AI inference sinks), and taint-tracking configuration.ai_inference_actions.model.yml: 15actionsSinkModelentries with kindai-inferencecovering GitHub official (actions/ai-inference,github/ai-moderator), Anthropic (anthropics/claude-code-action), Google (google-github-actions/run-gemini-cli,google/gemini-code-assist-action), OpenAI, Warp, and community AI actions.Variant analysis results
Validated through MRVA across 479 CodeQL
actionsdatabases and local analysis on 18 databases:anthropics/claude-code-actionoutput interpolated into shell commands)Test coverage
References