CCC.GenAI.TH04: Insecure / Unreliable Model Output
Threat ID:CCC.GenAI.TH04
Title:Insecure / Unreliable Model Output
Description:
A GenAI model may generate content that is incorrect, misleading or harmful, such as convincing misinformation (hallucinations) or vulnerable or malicious code, due to its reliance on statistical patterns rather than factual understanding. Directly using this flawed output without validation can lead to system compromises, poor decision-making, and legal or reputational damage.
Related Capabilities
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.F03 | Embedding Model Selection | Ability to select a foundation model used for tasks like semantic search, clustering, and document similarity by converting text into vector embeddings. |
| CCC.GenAI.F06 | Customizable Model Selection | Provide users the ability to fine-tune models with their own data. |
| CCC.GenAI.F07 | Parameter Tuning - Temperature | Ability to control the randomness and creativity of the response. |
| CCC.GenAI.F08 | Parameter Tuning - Max Token | Ability to limit the length of the response. |
| CCC.GenAI.F09 | Parameter Tuning - Top P (Nucleus Sampling) | Ability to adjust the number of likely next tokens to consider based on cumulative probability. |
| CCC.GenAI.F10 | Parameter Tuning - Top K | Ability to limit the number of token choices for the next word. |
| CCC.GenAI.F11 | Parameter Tuning - Stop Sequences | Ability to halt generation when a predefined sequence is encountered. |
| CCC.GenAI.F12 | Parameter Tuning - Frequency Penalty | Ability to penalize words that have been used frequently, reducing their likelihood of being repeated. |
| CCC.GenAI.F13 | Parameter Tuning - Presence Penalty | Ability to penalize tokens that have already been used, encouraging the model to introduce new tokens. |
| CCC.GenAI.F14 | Parameter Tuning - Context Length | Ability to control how much prior conversation or input the model will use for generating coherent responses. |
| CCC.GenAI.F21 | Generate Content | Ability to generate a response given a foundation model, parameter values, and a prompt. |
| CCC.GenAI.F25 | Plugin Integrations | Ability for the model to use tools to complete a model interaction. For example web search, python code execution or external maths engine. |
External Mappings
| Reference ID | Entry ID | Strength | Remarks |
|---|---|---|---|
FINOS-AIGF | AIR-OP-004 | 0 | Hallucination and Inaccurate Outputs |
FINOS-AIGF | AIR-OP-006 | 0 | Non-Deterministic Behaviour |
FINOS-AIGF | AIR-OP-020 | 0 | Reputational Risk |
SAIF | IMO | 0 | Insecure Model Output |
OWASP-LLM-TOP10 | LLM09:2025 | 0 | Misinformation |
OWASP-LLM-TOP10 | LLM05:2025 | 0 | Improper Output Handling |
MITRE-ATLAS | AML.T0062 | 0 | Discover LLM Hallucinations |
MITRE-ATLAS | AML.T0060 | 0 | Publish Hallucinated Entities |
MITRE-ATLAS | AML.T0077 | 0 | LLM Response Rendering |
Controls
| ID | Title | Objective | Control Family | Threat Mappings | Guideline Mappings | Assessment Requirements |
|---|---|---|---|---|---|---|
| CCC.GenAI.C02 | Model Output Filtering and Sanitisation | Inspect and validate GenAI model output before passing it to users, applications or plugins in order to filter or sanitise insecure or unreliable output and prevent sensitive data leakage. | Data | 5 | 7 | 2 |
| CCC.GenAI.C05 | Citations and Source Traceability | Require the GenAI system to provide citations or direct links back to the source documents used to generate a response, in to enhance the transparency, trustworthiness, and verifiability of AI-generated content. | Data | 2 | 1 | 1 |
| CCC.GenAI.C08 | Quality Control and Red Teaming | Establish a formal program for quality evaluation and adversarial testing (red teaming) to ensure GenAI system meet all business, quality, security and compliance requirements before getting deployed into production environments. | Model Assurance and Evaluation | 5 | 5 | 2 |