CCC.GenAI.TH06: Unintended Action by a Model-Based Agent
Threat ID:CCC.GenAI.TH06
Title:Unintended Action by a Model-Based Agent
Description:
A model-based agent, given the authority to execute tools or interact with APIs, may perform an action that is harmful, incorrect, or not aligned with the user's true intent in response to a prompt. This can be caused by the model misinterpreting an ambiguous prompt or being manipulated by an adversary into misusing its delegated authority.
Related Capabilities
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.F21 | Generate Content | Ability to generate a response given a foundation model, parameter values, and a prompt. |
External Mappings
| Reference ID | Entry ID | Strength | Remarks |
|---|---|---|---|
FINOS-AIGF | AIR-OP-018 | 0 | Model Overreach / Expanded Use |
SAIF | RA | 0 | Rogue Actions |
OWASP-LLM-TOP10 | LLM06:2025 | 0 | Excessive Agency |
MITRE-ATLAS | AML.T0065 | 0 | LLM Prompt Crafting |
Controls
| ID | Title | Objective | Control Family | Threat Mappings | Guideline Mappings | Assessment Requirements |
|---|---|---|---|---|---|---|
| CCC.GenAI.C02 | Model Output Filtering and Sanitisation | Inspect and validate GenAI model output before passing it to users, applications or plugins in order to filter or sanitise insecure or unreliable output and prevent sensitive data leakage. | Data | 5 | 7 | 2 |
| CCC.GenAI.C06 | Least Privilege for Plugins | Restricts the permissions of any external tools the GenAI system can call to limit the potential damage if an agent is coerced to perform unintended actions or vulnerabilities in the tools are exploited. | Identity and Access Management | 2 | 1 | 1 |