Ten is Less than Five

by Michael Camilleri

Almost exactly one year ago, John Gruber criticised the tendency of reviewers to grade to a curve in the context of tablets. Unfortunately, it’s been a year and things haven’t changed much.

Consider Vlad Savov’s recent review of the Sony Xperia S for The Verge.1 Savov gives the device a 7.1 making you think this might be a phone worth getting. Not the best of the best, but something you wouldn’t feel bad about buying. However, in the slick video2 that accompanies the review, Savov signs off by saying:

Overall, the Xperia S is a well-spec’d, well performing device that seems like a phone that could have been released three months ago. It has the same processor, display specs and base operating system as the HTC Rezound but comes so much later. I can’t advise anyone to go for the Xperia S today when the HTC One X and One S are set to arrive in early April with Ice Cream Sandwich, far better processors and, some might argue, even prettier design.

(Similar comments are made in the wrap-up of the text of the review itself.)

So we have a comment that this phone cannot be recommended for any one and then a score over 7? Sony isn’t a 5-year-old finger painting for the first time, Vlad. If their product is not one that you can recommend, then it should be by definition not a good product.

Would Savov have copped a lot of shit from Sony fans if he’d given the Xperia S a score of, say, 2? Probably. Worse, would he have been cut off by Sony PR from receiving future review units? I wouldn’t be surprised. But you know what the alternative has produced? A situation in which I no longer trust Savov’s reviews.

The good news is that I have a simple solution for The Verge: change your scale. Instead of using one out of 10, make it 5 (and only use whole numbers).3 Having a far narrower range of possible scores will focus the mind and help you to avoid this type of issue in the future. Does anyone think that if Savov had only the numbers from 1 to 5 available to him he would selected 4? (7.1 divided by two and then rounded up.) He would have at the very least gone 3 and, if he’d been game, maybe even 2.

  1. While I’m singling out The Verge (and Savov in particular) it must be said that The Verge is one of the best technology blogs to have come along in a very long time. The design is some of the best on the web, the attribution is ethical and the writing is easily above the average for tech writers (which may sound like I’m damning with faint praise but I mean that as a compliment). 

  2. I am not the first to say this but it must be acknowledged that The Verge’s video reviews are absolutely top-notch. Their entire production team deserves congratulation. 

  3. If you have real cajones, just use ‘Buy’ or ‘Don’t Buy’ (since this is the decision that consumers actually have to make).