Skip to main content

AI/ML / Multi Agent Refarch / Controls / DEV

AI System Observability

CCC.MARefArc.CN17 · DET

Instrument every layer to emit logs, traces, metrics, and events to the Observability Layer so that behaviour, drift, availability, and data handling are continuously visible and auditable.

Related Capabilities

IDTitleDescription
CCC.MARefArc.CP12Authoritative knowledge source basesInternal and external repositories of structured data, unstructured documents, and graph-based representations that provide authoritative information for grounding.
CCC.MARefArc.CP13Vector-based semantic retrievalVector databases providing semantic search and grounding so agents can find relevant information from large text corpora.
CCC.MARefArc.CP14Approved-model registry and lifecycleCatalog of approved models with metadata, version information, configuration parameters, and usage constraints, ensuring agents access only models meeting organizational, regulatory, and security standards.
CCC.MARefArc.CP16Model-interaction zero-trust guardrailsEnforces authentication and authorization for every inference request and applies input validation against prompt injection, output filtering and redaction, access control, rate limits, and cost management before and after model execution.
CCC.MARefArc.CP06Agent collaboration and orchestration patternsSupports supervisor/worker decomposition, skills-based routing, and agent-as-a-tool handoff for decomposing and executing complex tasks across multiple agents.
CCC.MARefArc.CP15LLM inference gateway routingValidates inference requests and routes each to the correct model instance, abstracting model hosting behind a consistent interface.
CCC.MARefArc.CP20Feedback engineCollects and aggregates structured and unstructured feedback from users, evaluators, and automated systems, including correctness assessments, preference signals, and quality ratings, to inform system improvement.
CCC.MARefArc.CP03Agent registry and lifecycle managementCatalog of available agents with their capabilities, metadata, and configuration, supporting versioning, lifecycle management, and controlled onboarding of new agents.

Related Threats

IDTitleDescription
CCC.MARefArc.TH22Poor-quality, drifting, and bias-amplifying dataInaccurate, incomplete, outdated, or biased grounding and training data lead to unreliable outputs, while data and concept drift erodes predictive power over time and amplifies historical errors at scale.
CCC.MARefArc.TH01Model memorization leaks sensitive data across sessionsThe hosted models accessed through the LLM layer may memorize sensitive inputs or training data and later disclose customer PII, proprietary algorithms, or trading strategies, including cross-user leakage into unrelated sessions.
CCC.MARefArc.TH02Hosted-provider data-handling exposureSensitive data submitted through the LLM gateway to third-party hosted models is exposed when the provider lacks transparent encryption, retention limits, or secure-deletion guarantees, leaving the institution without control over data it no longer holds.
CCC.MARefArc.TH08Denial of Wallet via token-expensive or unthrottled agentic callsToken-expensive prompts, large-document chunking, or poorly throttled agentic loops drive excessive model and tool invocations, exhausting token budgets, triggering throttling, or inflating cost beyond capacity planning.
CCC.MARefArc.TH09Technology service provider outage or degradationTight coupling to a specific external model provider with limited failover leaves the system exposed to provider outages or performance degradation under load, violating business-continuity expectations.
CCC.MARefArc.TH10VRAM exhaustion on model-serving infrastructureConfiguration changes, aggressive caching, or memory leaks in model-serving libraries behind the LLM gateway exhaust GPU VRAM, degrading responsiveness or crashing model serving.
CCC.MARefArc.TH19Silent model version, prompt, and deployment driftProviders silently retrain, re-prompt, or re-architect models, or change deployment and API defaults, shifting behaviour even when inputs are unchanged; without version pinning in the model registry this breaks reproducibility and validated behaviour.
CCC.MARefArc.TH17Non-deterministic and non-reproducible outputsProbabilistic sampling, internal-state variation, context sensitivity, and decoding parameters cause identical inputs to yield different outputs across runs, undermining testing, reproducibility, and reliable evaluation.
CCC.MARefArc.TH18RAG grounding failuresEven with retrieval, responses may contradict retrieved documents, drop caveats truncated by the context window, fill gaps with incorrect general knowledge, exceed authorized advisory scope, or adopt an inappropriate tone or certainty for the domain.
CCC.MARefArc.TH14Model overreach and scope creep beyond validated useAgents are used beyond their validated scope as users discover new applications or systems are repurposed without re-evaluation, producing unreliable outputs in untested contexts; weak registry scoping and orchestration boundaries accelerate the drift.

Assessment Requirements

IDTextApplicability
CCC.MARefArc.CN17.AR01Every layer MUST emit logs, traces, metrics, and events covering requests, model interactions, retrievals, and policy decisions to the Observability Layer.tlp-clear, tlp-green, tlp-amber, tlp-red
CCC.MARefArc.CN17.AR02Telemetry MUST be retained and queryable for the period required to support audit and incident investigation.tlp-clear, tlp-green, tlp-amber, tlp-red

Guideline Mappings

FrameworkIDRemarks
finos-airAIR-DET-004