The inconvenient truth about data driven decision making

We are all creatures of habit and learning. In this guest post by Vijay Vijayasankar questions whether we really do know how to measure and interpret the right things. It’s a good question at a time when so many are relying upon data driven decision making as a mantra to slavishly follow.

In business, metrics are things you worry about all the time. Over time, most people do come to a conclusion that not all metrics are helpful. In fact, no metric is useful if you just watch it – you need to act on it, measure again and so on . But do we take the time and effort to question the so called ‘useful’ metrics ?

All through my career, I have had to deal with Billings, Bookings and Backlogs (the three B’s – or BBB as commonly known), and I am now pretty sure that these are not very useful metrics like I used to think.

In my time as an analytics consultant, I have helped design and deploy several BBB solutions for my customers – big and small, and in three continents. As a lay investor, I am always trying to find ways of comparing companies to figure out where best to put my hard earned dollars. And most recently, when I had to evaluate potential employers, I had to deal with it yet again .

At an elementary level – bookings is the net total of all orders you have, billings is money you expect from customers for net fulfilled orders, and backlog is what remains unfulfilled , but you are sure of converting to billings. Unfortunately – it is not that simple or uniformly defined across companies .

Billings – or revenue – is (mostly) reliable across public companies . They have to follow strict rules. But the non public companies don’t disclose revenue – and that made it harder for me in comparing potential employers.

Bookings are the most arbitrarily defined. The problem is that the time dimension makes it hard to define. If you sign a deal with a customer that pays you $100K every year for three years for meeting certain milestones, do you call it $300K bookings? What happens if the milestone gets missed ? Given each contract is individually negotiated – this makes it very very hard at a company level, and impossible to meaningfully compare across companies. Add to that the confusion of non public companies who use billings and bookings interchangeably .

Even within a company , it can get hard to manage business by bookings. When you have many products to sell, and you bundle stuff with complex discounts to make ‘elephant killer’ deals – there is some inevitable peanut buttering that goes with it on management reporting . Essentially you can’t make serious decisions based on bookings alone. This gets very complex on budgeting, sales compensation and many other areas. But as creatures of habit – managers still do it.

Then there is the backlog. Backlog is hard to keep on top of – there are all kinds of things like write offs , return orders etc that affect backlog . A complex portfolio and organizational structure also makes it harder to use backlog as a management reporting tool.

There are many other things that affect BBB reporting that I haven’t mentioned – it could easily fill a book. But it raises an important question – can we make good decisions based on BBB?

Some CXOs use additional metrics – like number of downloads of software, number of comments in internet, number of jobs open in the market to augment things like BBB reports. All of those approaches come with side effects too.

  • Downloads – is it because software is great or because you had to issue a hundred patches ?
  • Comments – quantity of comments might mean good or bad, and sentiment analysis is still fairly a new discipline
  • Number of jobs in market – is it because software is great? Or is it an indication that the company is high maintenance ?

Does it mean you should not bother with measurements ? Absolutely not – you should of course measure everything that matters . But do so knowing all the limitations and nuances. And never trust one metric in isolation.

One final thought – good decisions are based on stuff that can be measured , but also on some important stuff that cannot be measured. For example – Trust. No one will argue that Trust is not a factor in any meaningful business decision. But it does not have a clean metric to measure. Just because something can be measured doesn’t increase it’s ability to make a decision better . Remember that when you run into the next person who pontificates mindlessly on some version of ‘data driven decision making!’

What do you think? Am I being too cynical from battle weary experience or touching on an ‘inconvenient truth?’

Featured image credit: © Olivier Le Moal – Fotolia.com

Vijay Vijayasankar

Vijay Vijayasankar

Vijay Vijayasankar is an experienced software and services executive , currently focused on the fascinating world of big data as VP channels at MongoDB. Vijay was Global VP of a big data at SAP , and prior to that was the head of forward engineering at IBM Global Business Services . Vijay is a mechanical engineer by training and holds an MBA . He can be found on twitter as @vijayasankarv and at his personal blog http://andvijaysays.com
Vijay Vijayasankar

@vijayasankarv

Husband , dad to 1 human and 3 fur kids , VP Global Channels business development @MongoDB. Tweets are personal views only & Retweets do not mean endorsement .
Vijay Vijayasankar
Vijay Vijayasankar

Latest posts by Vijay Vijayasankar (see all)

10 comments
cochesdiez
cochesdiez

I feel that often times we use data points just to back ourselves up. It was in my early work life where I asked a senior executive of a TelCo why they paid all these strategy consultants, for sure they had they knowledge already inside. The answer was simple. "In case something goes wrong I can say I have chosen the best". For me many of the measurements and atomised views we do today are more to feel safe and to say "I did what was in my best knowledge". 

Do we really have the time and the insights to make use of the information provided ?

praba01
praba01

Hi Vijay/Dennis,

I watched SAP's virtual press conference y'day. I think what I observed is related to this blog. Vishal Sikka mentioned scientists/researchers from Stanford  and other places analyzed 125 variants of 629 people & immediately concluded(preliminary conclusion) that the risk of type-2 diabetes is related to where we come from. He went on to say that people from East Asia region have much lower risk of Type-2 diabetes than people from America & so forth. He also alluded that our perception -- that the risk factors for Type-2 diabetes is related to food, exercise, diet, lifestyle etc. -- is wrong. In my opinion, the study answered who is riskier and not why. (Just like your example Vijay, why more downloads?) "Why" could still be food, exercise, diet, exercise etc.

More interesting is the fact their preliminary conclusion is based on one variant rather than 125 variants. If HANA is revealing something completely different from what I understood y'day or what we know about Type-2 diabetes, then it's definitely a great news for all of us.

You can Watch the full replay of the SAP HANA in the Cloud announcement event here: http://events.sap.com/hanalaunch/en/session/9167

His discussion on type-2 diabetes starts at 34m45s.

Regards,

Bala Prabahar



RichardDuffy
RichardDuffy

Theres a trade off here I think as many folks use data to justify a position they have taken after they have taken it.


In his book Blink, Malcolm Gladwell refers to the value of intuition or gut feel when making important decisions.


So I  believe that the key here is balancing the facts with the intangibles - either data or intuition on their own can lead to the wrong outcomes or decisions...the answer is to surround yourself with data, the tools to interpret it and people with experience to add commentary and as Tammy says - context.

tpowlas
tpowlas

I think you should also look at the context of the metrics - similar to what you said on a Big Data webinar - context is important.  


For myself I haven't had to look at the metrics you mention above for years.


I agree with your trust statement.  If a consultant has promised to deliver x by a certain week, doesn't reply or respond, would I want to do business with this consultant again?  Integrity and ethics are very important, and I would say override any of the metrics you list above.

dahowlett
dahowlett moderator

@tpowlas  Tammy - I sense you've hit on a nerve with which many of us are familiar: the good ol' passive-aggressive non-responsive tier 2 and 3 managers who don't want to know about whatever it is they feel threatened by but which will kill them in the long run.