...everyone who looked at my Web site yesterday recognized that the out of focus picture was of elephants. (If you haven't seen the April Fool's page, I've left it on the site, you can access it here.) But think about what is implied by this: it takes very little actual information for the brain to form conclusions. Indeed, hold your fingers over the image so you can't see the ears and you'll still recognize it as an elephant, only you'll probably think you're looking at the back end. (By the way, this is a real photo I took, only cropped. Darned that Olympus E-P1 autofocus system...)
I've written about this before: we're easily fooled by our own brains. Quick: how many megapixels does your camera really need to have if two vauge brown flaps are all that it takes to say "elephant"? This "fooling" works both ways, actually. If all we have is 6mp worth of information, our brain will fill in information as long as we don't do anything to call attention to the lower detail level. And if we have 24mp, we can look at our images close enough these days that we clearly see that something's missing, so we demand more. Ironic, isn't it?
This makes evaluating images difficult. You can't let your brain fill in missing detail or let it start asking for more. You need to assess the image you have, and only the image you have.
What I want is a "speed evaluator." Instead of looking at images manually in my browsing program (usually Photo Mechanic or Lightroom) and assigning the star ratings after careful consideration, I want something that'll throw the images up automatically for very short periods of time in random order (and repeating them at least three times). If I rate the same photo consistently, I trust that rating. If I rate it inconsistently, I want to take time to evaluate why that might be. Moreover, I want a room where I can put a dozen other people all with a rating control doing the same thing with my images. I then want to compare their ratings with mine.