Learning to Live With False Alarms
- Chris Drummond, Institute for Information Technology, National Research Council Canada
- Robert Holte, Department of Computing Science, University of Alberta
Anomalies are rare events. For anomaly detection, severe class imbalance is the
norm. Although there has been much research into imbalanced classes, there are surprisingly
few examples of dealing with severe imbalance. Alternative performance measures
have superseded error rate, or accuracy, for algorithm comparison. But whatever
their other merits, they tend to obscure the severe imbalance problem. We use the
relative cost reduction of a classifier over a trivial classifier that chooses the less costly
class. We show that for applications that are inherently noisy there is a limit to the cost
reduction achievable. Even a Bayes optimal classifier has a vanishingly small reduction
in costs as imbalance increases. If events are rare and not too costly, the unpalatable
conclusion is that our learning algorithms can do little. If the events have a higher cost
then a large number of false alarms must be tolerated, even if the end user finds that
undesirable.
Citation
C. Drummond,
R. Holte.
"Learning to Live With False Alarms". Workshop on "Data Mining Methods for Anomaly Detection", January 2005.
| Keywords: |
false alarms, machine learning |
| Category: |
In Workshop |
BibTeX
@misc{Drummond+Holte:05,
author = {Chris Drummond and Robert Holte},
title = {Learning to Live With False Alarms},
booktitle = {Workshop on "Data Mining Methods for Anomaly Detection"},
year = 2005,
}Last Updated: June 07, 2007
Submitted by Staurt H. Johnson 
