Knowing When to Stop
2 comment(s)
I recently had an interesting conversation with a fraud manager at a mid-sized ($2B assets) bank. We were discussing the accuracy of fraud detection systems, and everyone's favorite (or, better said, least favorite) topic – false positives. The particular frustration was how to deal with hundreds or even thousands of daily alerts, most of them false positives. There is a particularly acute problem at smaller institutions, with limited resources to review fraud alerts. The heart of our discussion boiled down to this question: when dealt a queue of fraud alerts for review, how do you know when to stop?
As with anything complex, there is no one right answer to this question. But my discussions with fraud managers tell me that many institutions use one of the following methods to determine "when to stop":
- When they get through the entire list. This is essentially throwing manpower at the problem by building a team that's large enough to get through all of the alerts generated by the system. They are at the mercy of the accuracy (or inaccuracy) of the detection system.
- When they run out of time. This involves allocating a set amount of time (e.g., 8am – Noon every day), and then getting through as many alerts as possible in that time window. They are at the mercy of the clock, and some ability to prioritize alerts.
- When it feels right. Basically this involves setting an arbitrary number of alerts to review based on experience, intuition or some combination. They are at the mercy of their gut feel.
I think there's another way to do this: stop when the cost of avoiding loss is greater that than the loss itself. Seems obvious, but let's think about it operationally.
Finding this breakeven point requires being able to accurately risk rank fraud alerts. Today, many fraud systems score and rank alerts – typically based on the dollar value of the item in question. However, very few of them generate scores that are actually correlated to fraud and potential loss.
Surprising, perhaps, but true.
This reality forces fraud teams to review all of the alerts produced by the system in order to find all of the possible fraud. This, by definition, creates a huge false positive problem. But, if you had a fraud detection system that produces risk scores that are tightly correlated to fraud and potential losses, you now have a much better sense of when to stop. And if you know when to stop, you know exactly what staff and resources you need to profitably handle the "optimal" volume of alerts dictated by the stopping point. You can also then put a value on the trade-off of reviewing more or fewer alerts than the optimal number, depending on other factors that are harder to quantify.
I like this method because it frames fraud prevention as a business issue (cost vs. benefit), not just as a cost of doing business, or a budget to be managed. Furthermore, it relies on economics to determine a rational stopping point and staffing model, versus being forced to stop by arbitrary manpower limits, time or gut feel. The crucial piece, of course, is making sure you have a fraud detection system that correlates risk score to actual fraud let me know if you'd like to learn more.
How do you determine when to stop reviewing fraud alerts? Does it work? How do you know?