We Trained a Skin analysis AI Model on Millions of Real Photos — What Actually Works in Production

Over the past few years, we’ve been building a mobile-first AI skin analysis system used by +1.000.000 users worldwide (except the USA and Canada). Unlike most research setups, this system operates on real-world smartphone images — not clinical data, but noisy, user-generated photos taken in uncontrolled conditions. To date, we’ve processed millions of images, with a curated subset of a few hundred thousand used for training. A fixed validation set of ~27,000 real-world images has been used to track performance consistently across model versions. This article isn’t about building a model from scratch. It’s about what actually works when you try to improve one in production — over years, not weeks. 1. A Fixed Validation Set Is More Valuable Than a Bigger One One of the most important decisions we made was also one of the least exciting. We stopped updating our validation dataset. Every model version was evaluated on the same ~27k real-world images. No rebalancing, no cleaning, no improv