Reviews - star ratings - averages

[Dave2002]

I bought two units of a product on two occasions - not too far apart in time, from am.uk.

One unit was rubbish, or at least not suitable for the application, so I wrote a one star review and returned the item.
The other unit seemed OK, so I wrote a more positive review, and kept the item. As this was for a different order I thought I might just be allowed two different reviews.

Amazon has now removed the first review - but that wasn't really my intention at all. I could modify the now misleading positive review by downgrading it to a mediocre rating of two or three stars, but that would also be a misrepresentation. The application circumstances were slightly different, so one unit deserved a very low rating, while the other unit, which actually worked for the intended purpose, deserved a significantly higher one. Not everyone reads all the different star reviews, or even a sample of the reviews at different star ratings. Probably very few people read the average (3 star) reviews. The point is that a rating may be only a very crude measure of satisfaction.

For CDs, which I suppose some round here might like, it is not uncommon for a CD to be rated highly for performance, but (particularly if it is a "historic" recording) much lower for sound quality. That's OK, and much more helpful than simply averaging the scores and "giving it foiv".

Another example of where averaging doesn't work well was an example of an investigation into which height of seat lorry drivers preferred. It turned out that there were two popular seat heights - so designers then produced lorries with seats which were at the mean of the two heights. Those seats turned out to not be popular with very many drivers. Using arithmetic means for bimodal (or even multi-modal) distributions just does not make sense, yet customer rating tools must often force people to make such "judgements" when evaluating products if they are just based on one scale.

[Pulcinella]

I've always thought that five-star ratings are skewed, as many people will opt for the middle option if reasonably satisfied, but 3 out of 5 is better than that! There is the 'no-star' option of course, but how often is that used?
That's why, in better evaluations, at least when a response is required, an even number of options can give a statistically more reliable and meaningful (no pun intended) result, as a more conscious choice has to be made to tick the right box.

[Dave2002]

I agree, but there are still situations where the same reviewer might want to give different ratings.

There are many problems with reviews and rating. One is where there are incentives to fill in a survey. Very occasionally I was asked to administer surveys like that. Basically there's a reward (possibly small ... maybe not - for example an Amazon voucher) for filling in a survey and giving possibly detailed views on some matters of interest to those doing the survey. However, there might be a time conflict, in that filling in the survey detracts from some other activity. I have been known to point out that under those circumstances, since the reward is usually based on a random selection, and not affected by the quality of the answers, that filling in the survey with random numbers is the quickest way to get it done - though obviously it will introduce a significant amount of noise into the data. I may have used that approach myself for some surveys I was persuaded to fill in - though in fairness if I care about issues I do normally try a bit harder than that, and if I don't I refuse to do such a survey.