Model overreliance and misplaced implicit trust in the output of a GenAI model may lead to the acceptance of inaccurate, biased or insecure outputs without proper validation or oversight, potentially resulting in operational failueres, compliance breaches and flawed decision making.
AI/ML / Gen AI / Threats / DEV
Model Overreliance
CCC.GenAI.TH05
Related Capabilities
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.CP21 | Generate Content | Ability to generate a response given a foundation model, parameter values, and a prompt. |
Related Controls
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.CN02 | Model Output Filtering and Sanitisation | Inspect and validate GenAI model output before passing it to users, applications or plugins in order to filter or sanitise insecure or unreliable output and prevent sensitive data leakage. |
External Mappings
| Framework | ID | Remarks |
|---|---|---|
| FINOS-AIGF | AIR-OP-018 | Model Overreach / Expanded Use |
| FINOS-AIGF | AIR-RC-022 | Regulatory Compliance and Oversight |
| SAIF | IMO | Insecure Model Output |
| SAIF | RA | Rogue Actions |
| OWASP-LLM-TOP10 | LLM06:2025 | Excessive Agency |
| OWASP-LLM-TOP10 | LLM09 | Overreliance |
| MITRE-ATLAS | AML.T0062 | LLM Trusted Output Components Manipulation |