A risk management policy that lists "bias," "hallucination," and "data privacy" as risk categories isn't wrong, exactly — it's just not operational. Naming a risk isn't the same as having a process that catches it before it reaches a customer.

Regulated institutions already run mature risk management programs for credit risk, operational risk, and third-party vendor risk. The gap most teams run into with AI isn't a lack of risk management discipline — it's that AI-specific risks don't map cleanly onto existing categories, so they either get force-fit into a framework that doesn't quite address them, or handled ad hoc outside any framework at all.

The risk categories that actually apply to customer-facing AI

Four categories cover most of what goes wrong in practice:

  • Incorrect output risk. The system gives a wrong or misleading answer — the most obvious risk, and the one most vendors focus their marketing on mitigating.
  • Inconsistent output risk. The system gives different answers to the same question depending on wording, time of day, or which model version happened to be live — often more damaging than being wrong outright, because it's harder to detect and defend.
  • Undocumented decision risk. The system makes a correct decision, but there's no record of why, which becomes a problem the moment anyone needs to explain it after the fact.
  • Scope creep risk. The system starts handling cases nobody explicitly authorized it to handle, because a broad model doesn't know the boundary of what it should decide versus what it happens to be capable of answering.

Notice that only the first category — being wrong — is what most AI evaluation focuses on. The other three are architectural risks, not accuracy risks, and they don't improve with a better model. They require a system designed to prevent them structurally.

Most AI risk isn't "the model was wrong." It's "the model was inconsistent, undocumented, or operating outside its intended scope" — and none of those improve with a bigger model.

A practical scoring approach

Rather than trying to score "AI risk" as one undifferentiated number, score each use case on two dimensions before deployment: decision reversibility (can a wrong outcome be easily corrected, or does it cause lasting harm — a declined claim versus a mistyped confirmation number) and rule clarity (is there an unambiguous, documented rule governing this decision, or does it require judgment). Plot any proposed use case on those two axes, and the deployment approach falls out naturally:

  • High rule clarity, low reversibility harm — safe for full automation with a deterministic core (e.g., routine balance inquiries).
  • High rule clarity, high reversibility harm — automate the decision but log and spot-check it (e.g., standard fee waivers).
  • Low rule clarity, any harm level — route to a human by default; automation here is where undocumented and scope-creep risk concentrate (e.g., hardship determinations, clinical judgment calls).

How this maps to the NIST AI Risk Management Framework

Institutions building or refreshing an AI risk program are increasingly asked whether it aligns with the NIST AI Risk Management Framework (AI RMF) — the voluntary federal framework that has become a de facto reference point for examiners and auditors even outside sectors where it's formally required. It organizes AI risk management into four functions, and it's worth showing how the practical approach above maps onto them:

  • Govern. The organizational structure and accountability behind AI risk decisions — this is the ownership and change-control apparatus described in our companion piece on AI governance for banks and health systems.
  • Map. Identifying the specific risks a given use case carries — this is the reversibility and rule-clarity scoring described above, applied before a use case ships.
  • Measure. Ongoing assessment of whether the system is performing as expected — the monitoring and drift-detection process that catches inconsistent or scope-creeping behavior after launch.
  • Manage. Acting on what Measure surfaces — re-scoring a use case, tightening a rule, or escalating more cases to a human when drift is detected.

An institution doesn't need to adopt NIST's exact terminology to satisfy an examiner asking about it — what matters is being able to show a functioning equivalent of each of the four functions, mapped explicitly if asked. Institutions that can draw this map quickly tend to fare much better in an exam than institutions that have good practices but have never organized them against a recognized framework.

Building this into a broader AI risk management framework

Everything above works as a standalone approach to scoring individual use cases, but it should ultimately live inside the institution's broader AI risk management framework — the same governing document that assigns ownership, sets change-control procedure, and defines the review cadence described in our governance article. Treating AI risk scoring as a one-off exercise rather than a standing part of that framework is the most common way institutions end up with good practices for their first few use cases and no consistent process by the time they have twenty.

Making the framework operational, not just documented

A scoring framework only reduces risk if it's actually applied before a use case ships, not retrofitted afterward to justify a decision already made. That means every new automated use case gets scored against these two dimensions as part of the deployment checklist, the scoring decision is documented and owned by someone specific, and any use case that later shows drift — inconsistent answers, edge cases slipping through — gets re-scored rather than patched with a prompt tweak.