My AI model hit 87% accuracy on a test set and I think that's a problem

I was training a new model for image classification on a dataset of 10,000 pictures. The popular view is that higher accuracy is always better, but when I saw the 87% result, it made me question if we're focusing on the wrong metric. What specific things should we be looking at instead of just chasing a higher number?

3 comments

3 Comments

martin.paige8d ago

Honestly that reminds me of when my friend's spam filter was "99% accurate" but kept letting the worst phishing emails through. What's your model messing up the most, the tricky edge cases?

fox.david7d ago

What's the point of 99% if that last 1% is the stuff that ruins your day?

miakelly19h ago

Ever look at which specific pictures it's getting wrong? I had a model that was great at obvious stuff but failed hard on anything with weird lighting or odd angles. Started tracking error types instead of just the overall score, and that showed me where to add more training data. Like @fox.david said, that last bit is what kills you. My fix was to go find a bunch of examples of the exact things it messed up and retrain on those.