The Met Office and how to make accurate data 'worse than useless'
- Summary:
- Simplistic symbols make the BBC's online weather forecast 'worse than useless' in the view of one user. Is the BBC, the Met Office or the casual user at fault?
So I was interested to learn of a friend's recent correspondence with the UK Meteorological Office about the symbols the BBC uses on its weather pages to summarize the day's weather. (He initially contacted the BBC, who forwarded his email directly to the Met Office for a reply).
The exchange of emails highlights several ways in which graphical representations of data are easily misunderstood. But does the fault lie with the data scientists who filter the information, or with the unrealistic expectations of users when they interpret the resulting visualization?
My friend had become exasperated by the sunny symbol chosen to represent a partially showery day in his home town of Bruton, Somerset, in early June (see screenshot above). As he explained to the Met Office:
You will see that when I captured the screen shot on Sunday, the symbol for Tuesday was blazing sunshine, indicating a sunny day and (to my mind at least) not a chance of rain. The detail and the words tell another story — namely 'a mix of sunny spells and further showers on Tuesday, perhaps heavy and thundery at times.' In fact, the hourly/three-hourly detail shows a cloudy morning with showers and sunny intervals, turning to sunshine in the afternoon. That is not conveyed at all by the full sun symbol chosen for the day as a whole. I would expect an occluded sun with a drop of rain as the symbol for the day.
The problem, he summarized, was this:
This is not an isolated incident — it happens quite a lot, which makes the daily symbol for all practical intents and purposes useless, as I have learned that I must always check the detail to know the true story.
This might have deserved the riposte that this is par for the course if you live in the southwest of England, where the weather is so changeable that it is impossible to summarize an entire day in a single symbol. Instead, the Met Office patiently explained the process by which the day's symbol is selected:
The day (and night) symbols are ... averages over the day and night hours respectively. Best Data knows when sunrise and sunset are, and it adds up all the hourly values that fall within these times to create a symbol for the day. Again it will be weighted towards more hazardous weather, but will also be governed by what the majority of the day will be like.
There was no explanation, by the way, or who or what 'Best Data' is or how it acquired its potentially hubristic name. The writer went on to add that, as the day progresses, this hidden automaton continues to recalculate the average based on the prediction for the remaining hours of the day (or night), with the result that the day symbol may change to reflect a shift in the average weather over the remaining time.
While this was a clear explanation, it did not address the main bone of contention, as my friend reiterated:
I do not believe that an overall daily symbol of full sun fairly reflects the forecast for the day. Surely the overall daily symbol needs to have a drop of rain in it to show the possibility of showers?
The Met Office admitted he had a point, and consulted one of its data scientists, who provided further elucidation:
In the day symbol we are trying to produce a one-glance summary of the day that is most useful to most people. There has to be a point where we discount rainfall when this is likely to be only a brief interruption to an otherwise fine day. This case will have been close to the tipping point in the logic between sunshine and a showery day symbol. A case can be made for either option. If we had indicated a showery day people who had not gone beyond the day symbol would have reasonably assumed a threat of rain all through the day which was no better a reflection of the forecast than that there was no chance of rain.
The crux of the problem becomes clear as we read between the lines of this reply. The data scientist explains that the analytics are primed to swap out one symbol for another at a certain tipping point. This confirms that the selected symbol accurately reflects the underlying data. But it still fails to address the question of how people might actually use the information to plan their day. It is accurate, but it is neither precise nor informative. As my friend replied:
I would also disagree that a shower and sun symbol would make people assume a threat of rain all through the day. The nature of showers is that they are transient. What it would do is alert the user to the need to take some rain protection if they were going to be out all day or to check the detailed forecast to find out when it was likely to be dry. A full sun symbol signally fails to do that.
In other words, the symbol interprets the data accurately but in a way that ignores the context in which it is useful to the end user. What I suspect is really happening here is that my friend's complaint is falling between two stools. It is the BBC's choice of symbols that is letting him down. But because the BBC has referred his complaint to the Met Office, they can only respond by explaining how they are meeting the terms of the contract they have with the BBC.
Related stories
- The day the iPlayer wasn't there highlights how much the BBC platform is part of the digital DNA (diginomica.com)
- The Weather Company forecasts change, deploys Google Apps (diginomica.com)
- Sorry becomes the easiest word for the BBC's doomed digital play (diginomica.com)
The only way to resolve this matter to his satisfaction would be to refer it back to the BBC to reconsider the range of symbols they use to illustrate various weather possibilities. But he is now corresponding with people at the Met Office that have no say in such strategic decisions (which is why I sought his permission to report the exchange here — perhaps now the right person will have their attention drawn to this problem. Are you listening BBC?)
But is his request reasonable or is he in fact expecting the BBC to convey an unrealistic degree of information in a single icon? When he reposted this email chain to our discussion group, another friend chimed in. She has professional expertise in how people interact with computers as well as a keen amateur interest in rock climbing. This was her take:
The Met Office overview symbols are, in my experience, pretty much useless, for all the reasons you list. I've learned to ignore them as they don't tell me the detail I need for deciding whether or where to go climbing (or whether to take a raincoat / suncream / whatever).
But I'd also say that the Met Office has a very difficult challenge in communicating the information that different people require for different purposes, accurately enough for people to base decisions on that information.
Perhaps the true failing of the BBC's daily weather icons is that they're not sufficiently aware of the context in which each of us are consulting them, and it should be able to tailor its iconography to the disparate needs of climbers, sailors, outdoor events organizers and those of us who want to know whether to bother with an umbrella. (Maybe there should be an app for that).
The broader moral of this story is that accurate interpretation of the underlying data is not the only consideration when deciding how to visually summarize information. The right presentation is also influenced by the context in which the results are going to be used. Get that wrong, and however accurate the data may be, users will reject it as "for all practical intents and purposes, useless."