Skewed social scoring for hotels

The other day I received the results of an intriguing study by Medallia that looks into the rankings of different hotel groups. The US only survey, which measures multi-property groups and is dated for the end of June 2013 ranks based upon the output of responses from a variety of booking sites. The sources are not revealed but you can bet that Booking.com, Expedia and TripAdvisor figured strongly in the results.

Medallia calls the report the 'Hotel social media report' as the eye catcher whereas I would call them 'aggregated review scores' and have done with it.

Comparisons are drawn with the previous year in order to provide a leader board. It divides the overall universe of results into luxury,upper upscale, upscale, upper midscale, midscale and economy which is certainly fair enough. Finally, there is limited commentary on the best and most improved brands within those segments. Why am I intrigued?

The statistical basis upon which the report is drawn is flawed. Check out the image which shows the results for 'upper upscale.' You might need to click the image to fully understand what I mean.

Five flaws

First up, the sample sizes vary wildly. You might expect that larger numbers would appear for brands like Hilton, Marriott, Sheraton and Hyatt, names which spring easily to mind. Equally interesting is an analysis from List Dose that ranks Marriott and Hilton Worldwide as 3 and 2 respectively as:

 hav[ing] been able to grab the eyeball of the visitors towards them

That in turn means there will be unequal weighting of results as between sample sizes. As far as I can tell, there has been no obvious effort to smooth out those differences.

The second problem is that at least some of the sites which encourage feedback have been criticized for adopting rating methods that leave a lot to be desired. Last year, TripAdvisor got into hot water for allowing people to rank without leaving comments. It ended up getting spanked by the UK's Advertising Standards Authority. Even now, I find that reviews on places like TripAdvisor need to be scrutinized carefully in order to come to a balanced assessment. Even then, I look for people I know leaving reviews above any random person.

Next, we know nothing about the kind of traveler making up the universe from which Medallia collected the data. Here, I think Suzanne Moore at The Guardian makes an astute observation when she says:

Ordinary people when given free rein are incredibly petty but mostly truthful. Such sites cut out the middle men or women, the professional "travellers" who never stay anywhere other than five-star or fly cattle-class. Write about what you did on your holidays? Yes we do, at great and pedantic length.

The excising of the expert reviewer is happening right across the board. Who needs expertise when every Tom, Dick and Harriet reviews everything for free anyway. Isn't this truly democratic? The nature of criticism is changing, so this hierarchy of expertise is crumbling.

My requirements when I travel for business are very different for when I travel for pleasure. Here, TripAdvisor (since I'm on their back) does a fair job of breaking down demographics.

Then there is the problem of understanding why people comment. We don't have a good answer for this but I find myself increasingly turned off by the pestering emails from review/booking sites and now the hotels themselves which routinely plead with me to say something, anything. My sense is that you have to be very determined or motivated to leave comments and rankings these days. that in itself brings into question the value of comments and rankings.

Finally, Medallia has fallen into the trap of lauding only the best of the best. Of itself, there is nothing wrong with that although I am now somewhat bored of seeing TripAdvisor hailed plaques seemingly springing up like weeds in hotel lobbies. We need to understand failure much more than success but I guess that doesn't fly so well.

The missing context

When I am making choices I want to know why one group is doing better than another and why another might be failing. I don't need chapter and verse, just the headlines. As an example, the other week I stayed in a well known and oft used spot in Palo Alto. It was a business trip and I was OK with what was on offer. But truth be known, the place has seen better days and while the facilities were adequate they weren't worth the rate I paid. The fact the place was deserted of an evening is telling. You get the snapshot picture. While on that point, I find that return traveler reviews are far more useful as they provide perspectives over time which will inevitably factor into rating changes. How do those get evaluated?

Medallia was trying to make a point about improving quality with the motivation that readers might be interested in their hospitality solutions. And of course the buyers' motivation must surely be about offering a more competitive service. If only that was really true.

When corporate event organizers routinely gobble up thousands of rooms in major cities around the world, adjust rates to suit pocket depth and demand for any particular day, and where they are only truly interested in that sparkling review, you have to ask: who is gaming who?

Verdict

The Medallia report should not be wholly written off. It is a good starting point for those interested in understanding service in the hosptitality business. But I see this as a microcosm of much that is awry with the use of social media - the source of much Big Data in the minds of many.

So while I equally congratulate Robin Fray Carey for curating her way to one million unique visitors per month visiting Social Media Today and counting more than 200,000 professionals in that community, I wish those same professionals would stop talking about 5 Reasons Your Brand Needs Social Customer Serviceand other lazy lists and start talking about meaning and context.

If social media is to become a reliable and useful resource for everyone then stepping away from convincing people of its value and applying rigor wouldn't go amiss. Basic stats disciplines would be a good start.

Image credit - © pixarno - Fotolia.com