← Back to Technical Section

GenAI Governance & Compliance

Risk management, observability, EU AI Act compliance, human-in-the-loop controls, and use case classification for enterprise GenAI systems.

Why Governance is Non-Negotiable

Generative AI systems can produce incorrect, biased, or harmful outputs; leak sensitive information through prompt injection; and make consequential decisions without human oversight. In regulated industries, the absence of governance is not a technical debt item — it is a legal liability. Frameworks like the EU AI Act and NIST AI RMF are now mandating documentation, transparency, and continuous monitoring for high-risk AI systems.

Regulatory Reality: The EU AI Act requires high-risk AI systems to maintain traceability of decisions, document training data and prompts, implement continuous monitoring, and provide human override mechanisms. Non-compliance carries fines of up to 3% of global annual revenue.

Risk Assessment Matrix

Architectural choices carry inherent trade-offs between flexibility, speed, cost, and governance burden. Understanding this matrix helps select the right approach per use case.

Architecture	Flexibility vs Simplicity	Speed vs Governance	Cost vs Performance
Deterministic Chain	High simplicity, low flexibility	Fast; minimal governance needed	Predictable cost; limited output quality
Multi-Agent	High flexibility; complex orchestration	Requires governance to avoid erratic behavior	Higher cost; optimizable with parallelism
Self-Managed	Maximum control; high dev effort	Internal governance required; full responsibility	Fixed costs; high upfront investment
Fully Managed (Bedrock)	Low ops effort; vendor dependency	Built-in policies and evaluators available	High per-token cost; scales without capex

Observability

You cannot govern what you cannot observe. Full telemetry across the agent execution graph is the foundation of every other governance practice.

Logs

Structured JSON logs per agent invocation: session ID, agent role, input/output token counts, tool calls made, latency, and critic scores.

Metrics

Token throughput, cost per session, agent latency P50/P95/P99, cache hit rate, critic pass rate, and error rates — exposed via CloudWatch or Prometheus.

Traces

End-to-end distributed traces using OpenTelemetry spans per agent, tool call, and LLM invocation. Essential for identifying orchestration bottlenecks.

Automatic Evaluators (Critics)

Critic agents or built-in evaluation functions score every agent output before it reaches the user or triggers downstream actions. This creates a continuous quality gate across the entire system.

Evaluation Dimensions

Factuality

Is the output grounded in retrieved context?

Policy Alignment

Does it comply with content and usage policies?

Completeness

Does it fully address the user's intent?

Safety

Does it avoid harmful or sensitive content?

Human-in-the-Loop Controls

For high-impact decisions — credit approvals, HR recommendations, medical triage, legal document generation — human oversight is not optional. Well-designed systems make human review efficient rather than burdensome.

Design Principle: Human review should be triggered by risk, not by default. Use automatic critics to filter low-risk outputs and escalate only those that fail quality thresholds or touch sensitive decision categories.

Risk-based escalation: Outputs with critic scores below threshold automatically route to human review queue.
Approval workflows: High-impact actions require explicit human approval before execution (e.g., Step Functions waitForTaskToken).
Audit trails: Every human decision — approve, reject, modify — is logged with reviewer identity, timestamp, and rationale.
Feedback loops: Human corrections feed back into evaluator training to improve automatic critic accuracy over time.

EU AI Act & NIST AI RMF

EU AI Act Requirements

Risk classification of AI systems (minimal, limited, high, unacceptable)
Technical documentation for high-risk systems
Transparency and explainability obligations
Human oversight mechanisms
Continuous post-market monitoring
Data governance and training data documentation

NIST AI RMF Actions

Govern: Establish policies and accountability
Map: Identify context, risks, and affected stakeholders
Measure: Evaluate and monitor AI risks continuously
Manage: Prioritize and respond to identified risks

Security Risks

Warning: Prompt injection — where malicious content in retrieved documents or user input hijacks agent behavior — is the most common attack vector in RAG and tool-using agents. Always sanitize retrieved content before injecting it into agent context.

Prompt injection: Attackers embed instructions in documents or user inputs that override system prompts.
Data leakage: LLMs may inadvertently reveal PII or confidential data from retrieved context in their responses.
Tool abuse: Agents with overly broad tool access can be manipulated into executing unintended actions.
Shadow AI spending: Uncontrolled model API usage by individuals bypasses both governance and budget controls.

Use Case Classification & Architecture Mapping

Category	Example	Pattern	Infrastructure	Risk Level
Simple Automation	Data extraction, notifications	Deterministic chain	Lambda / n8n	Low
AI-Assisted Workflows	FAQ bot, onboarding assistant	Single agent + RAG	Bedrock AgentCore	Medium
Multi-Agent Reasoning	Complex support, report gen	Orchestrator-worker + critics	Step Functions + Bedrock	High
Governed Enterprise	HR decisions, credit approval	Multi-agent + audit + HITL	AgentCore + Cedar + Audit DB	Very High

Compliance Checklist

Technical Controls

Structured audit logs per agent invocation
Automatic critic evaluators in production
Human review queue for high-risk outputs
Prompt injection mitigation
PII detection before logging
Minimal-privilege tool access per agent

Process & Documentation

Risk classification per use case
System prompt and model version registry
Data lineage for training and RAG documents
Incident response playbook for AI failures
Periodic bias and drift evaluations
Documented human override procedures

Test Your Knowledge

Score 8/10 or higher to pass