CCC.GenAI.TH05: Model Overreliance
Threat ID:CCC.GenAI.TH05
Title:Model Overreliance
Description:
Model overreliance and misplaced implicit trust in the output of a GenAI model may lead to the acceptance of inaccurate, biased or insecure outputs without proper validation or oversight, potentially resulting in operational failueres, compliance breaches and flawed decision making.
Related Capabilities
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.F21 | Generate Content | Ability to generate a response given a foundation model, parameter values, and a prompt. |
External Mappings
| Reference ID | Entry ID | Strength | Remarks |
|---|---|---|---|
FINOS-AIGF | AIR-OP-018 | 0 | Model Overreach / Expanded Use |
FINOS-AIGF | AIR-RC-022 | 0 | Regulatory Compliance and Oversight |
SAIF | IMO | 0 | Insecure Model Output |
SAIF | RA | 0 | Rogue Actions |
OWASP-LLM-TOP10 | LLM06:2025 | 0 | Excessive Agency |
OWASP-LLM-TOP10 | LLM09 | 0 | Overreliance |
MITRE-ATLAS | AML.T0062 | 0 | LLM Trusted Output Components Manipulation |
Controls
| ID | Title | Objective | Control Family | Threat Mappings | Guideline Mappings | Assessment Requirements |
|---|---|---|---|---|---|---|
| CCC.GenAI.C02 | Model Output Filtering and Sanitisation | Inspect and validate GenAI model output before passing it to users, applications or plugins in order to filter or sanitise insecure or unreliable output and prevent sensitive data leakage. | Data | 5 | 7 | 2 |