The difficulty in assessing the value of digital, according to the NAO

Profile picture for user ddpreez By Derek du Preez February 13, 2017
Max Tse, director at the NAO, explains that digital projects require a different approach to auditing, compared to big bang projects of the past.

For years the National Audit Office (NAO) has been feared amongst civil servants responsible for big change programmes in government. Get things wrong, lose money, don’t deliver, and the NAO will write a lengthy report on what went wrong and who is to blame.

The role of the NAO is to ensure that government departments are delivering value for taxpayers’ money.

However, as we increasingly see digital transformation take the centre stage in Whitehall, is the auditor’s job becoming more tricky? In the past the NAO was able to assess a comprehensive, pre-defined business case, which likely followed a project over a number of years, had a set budget and timeframe, and had very specific desired outcomes (often to save money).

But transformation projects in government now very may well not have such a strict ‘route to market’. Agile delivery requires a certain amount of ambiguity, allowing space for things to go wrong and/or change. Equally, it has been said that the government wants to move away from systems that have a ‘shelf life’ (e.g. save X amount of X number of years and then need to be reassessed), to a delivery model of continuous improvement.

How do you assess the value of continuous digital improvement?

Max Tse, director at the NAO, speaking at the Think Digital Government 2017 conference in London last week, discussed this exact quandary, where he said that there is “no playbook” for digital and that there is a fine balance to be struck.

Tse said:

When it comes to digital transformation, we are interested in two main questions. The first: is money being spent wisely? We all agree that the potential is enormous, but at the same time we have to pull that back to: how do we make that work in a constrained environment where money is tight.

And secondly, how do we understand, prioritise and choose between the different things we are doing? How do we think about this in a way that recognises the opportunities for innovation, so that we’re not constrained by imposing overly tight business cases - but at the same time don’t allow things to proliferate to the extent where we don’t understand what we are doing.

Ultimately, the NAO will need to take new considerations into account as it attempts to audit digital projects. That being said, Tse still, obviously, sees money playing a role.

I do think that we need to care about users and how the experience of users comes through. And we need to think about outcomes. I think we’ve always known this. I don’t think it’s a new problem for government. Actually, I think it’s a new opportunity.

I think we will need to reshape the way we think about these things. And we need to think broadly about it. Having said that there is always a question of money. And money is a very useful way of understanding where the value of things is delivered. Also, understanding the cost of things as we balance priorities.


Tse went on to say that the biggest challenge for government, and the NAO, at the moment is with regards to how it assesses the value of all its options. With so many possibilities on the table, especially when you consider the attempts to move away from large outsourcing agreements, and then adding Brexit to the mix, it isn’t possible to do everything.

Assessing all of the options, which won’t be delivered necessarily in a traditional way, is “incredibly difficult for government”.

Tse said that this is further complicated by the variety of projects undertaken across Whitehall. For example, the NAO is seeing the following:

  1. Departments undertaking tactical improvements, using digital, to improve currently flawed processes. Tse said that assessing the value of these projects is about understanding the proportionate investment. Should said department be spending money on tactical improvements or should it be looking at a more broad systemic change?
  2. The second example that the NAO is assessing is with regards to flexibility across government. This is particularly true of the work that is being done to improve the way that the private sector interacts with the public sector, including the work that has gone into improving how the government contracts work. Tse said that this has been hard for the Civil Service, as it isn’t used to spending lots of money up front to improve things in the future, for an outcome that isn’t defined.
  3. The third type of project the NAO is looking at is the large-scale transformation programmes, which Tse describes as “incredibly complex”, as they involve rethinking the policy, the technology, the process, and the business design.

Tse said:

I don’t think anybody has really got this solved. There is no playbook for just doing how transformation is going to work. I think what people have learnt through processes like Universal Credit is the importance of understanding the legacy environment that we are changing. I think it’s really encouraging that the Government Transformation Strategy recognises explicitly and puts a lot more emphasise on the underlying environment.

Business man with smartphone and digital waves © everythingpossible -
And the challenge rests in providing people with the flexibility to change tack if they need to, but within the confines of a structure that they can still be held to account. Tse said:

I think there are still a lot of transformation programmes that are currently ongoing and are struggling with this, trying to think about how to manage that. As an auditor, it is incredibly frustrating to look at the programme and say ‘well you said you’d do this by this point’ and then someone will tell you that the timescales and the cost have all changed. But you understand why and they have justifications.

How do we get that balance between holding to account and keeping focused on what was originally intended? And allowing the flexibility to pursue this ultimate user need and outcome in a way that isn’t overly constrained. There’s nothing worse than a programme ploughing along the same lines just because that’s the way that they spec’d it out, knowing that it’s wrong.”

What we’ve found is that as long as there is a very clear guiding mind at the centre of what people are trying to achieve, and they have an idea of what they’re trying to achieve and measure, it’s absolutely fine. I think where we do have a really difficult question is where the objective is slightly vaguer, and there are things that sound like they are taking you in the right direction, but really knowing if you are making progress is much harder.

My take

It’s a tough one. I’m not sure what the answer is. I agree with Tse that there is nothing worse than watching a department dogmatically pursue a project because of an unrealistic promise that was made by a politician - who likely has little idea about the technology or project implications.

As noted above, the whole idea behind agile is that you don’t necessarily know where you are heading when you start, and there is scope to change and make mistakes. Ultimately, the value can only really be assessed in the outcomes and the experience of the citizens. What does that mean for the role of the auditor? Who knows.

On one final note, Tse did have some positive words to say about GDS, which I think are worth leaving here. He said:

We are doing work at the moment actually looking at how GDS is positioned in the future. We recognise that it’s been a very fast changing environment. The role of GDS in challenging government, I don’t think we should underestimate that.

It really has challenged government to think differently about the way that technology is used and to adopt new approaches, which are now pretty much taken as part of the way that people do things. People at the top levels are talking about things that they never would have done.

Image credit - Change pollution with green environment © alphaspirit -; Business man with smartphone and digital waves © everythingpossible -

Disclosure - diginomica is a shareholder in Think Digital Partners, our official events partner.