A report from JPMorgan AI Research shows that picking the right fraud KPI can improve your fraud performance by at least 20%. In this article I’ll be evaluating the pros and cons of Value Detection Rate and Transaction Detection Rate.
There is always opportunity to do better in fraud prevention – stopping more fraud at a lower cost and less customer impact. That’s why measuring the performance of the fraud models, and in a wider sense, the fraud organization is important.
These Key Performance Indicators (KPI) will help you evaluate your models and fraud system as well as show the overall performance of the fraud organization.
What are the best fraud KPIs to use?
There are a number of KPIs that can be used to measure performance. Typically, the main list of KPIs include:
- Attempted fraud
- Captured fraud
- False positive ratio compared to true positive ratio
- Authentication step-ups
- Interventions and warnings
- Alert volume (both absolute and as a percentage of transactions)
- Number of calls per operator per hour
- Cost of call center operator per hour
- Ratio of automated alert handling compared to manual alert handling
- Recovery of funds (net loss vs gross loss)
- Etc.
I often get asked as to what the right KPI to enable a fair and accurate measurement of the fraud performance is. But the truth is that every organization has different approaches. Whether they focus on fraud losses, customer impact, handling of fraud, operational costs and/or compliance risks.
What is important is to consider what is the best way measure performance without causing unintended side-effects and provide a true reflection of the performance.
Three core fraud capture KPIs
If we zoom in specifically on how to measure the fraud captured, there are typically three ways to characterize the fraud captured:
Transaction Detection Rate (TDR)
Transaction Detection Rate (TDR) is the number of fraudulent transactions that was detected by a monitoring system, as a percentage of the total number of fraud transactions. In academic articles the very similar measure of P@k (Precision-at-k) is often used, which further highlights how this depends on the monitoring system parameter “k”, which is the model threshold value.
Account Detection Rate (ADR)
Account Detection Rate (ADR) is the number of accounts or users that was detected by a monitoring system, as a percentage of the total number of accounts/users which encountered fraud.
Value Detection Rate (VDR)
Value Detection Rate (VDR) is the transaction value of all the transactions that was detected by the monitoring system, as a percentage of the total value of all fraudulent transactions.
The actual level of these detection rates varies depending on strategy, attempted fraud, operational capacity, etc. which is why it always needs to be put in the context of the other fraud KPIs.
What fraud capture KPI should you use?
A scientific study from 2023 look at how the choice of TDR vs VDR effected the fraud losses. It confirmed that VDR provides at least 20% more reduction in fraud losses, compared to measuring TDR.
While it’s worth noting that the measurement was made on fraud data for debit and credit cards so there might some variation for account-to-account payments. This research highlights that the choice between these KPIs makes a significant difference.
While it might be more complicated to measure VDR in all cases, this shows the impact if you settle for ADR or TDR. Many academic reports are based on TDR statistics, or the similar measure P@k and I hope we get to see more studies focused on evaluating VDR as a measure.
It is important to note that TDR and ADR are still useful metrics. For example, ADR aligns closer with the customer experience impact of fraud as it better captures the number of customers impacted. Similarly, TDR provides a closer alignment to speed of detection and detection of tester transactions.
What is important is that you consider the goals of your organization and department and select the measures that most appropriately align with these.
Please do get in touch if you’d like to discuss how you can determine the appropriate goal(s) for your team and what tools are available to help you do so.
More information is available in the scientific study here: Danial Dervovic, Saeid Amiri, Michael Cashmore. Value Detection Rate: A Performance Metric for Payments Fraud Detection, JP Morgan AI Research, 2023
