How Fannie Mae uses CAST to crack the Curate’s Egg

Computer-Assisted Software Testing can turn out to be something of a 'Curate’s Egg' for some enterprises. The 'good' parts are the ability to scan applications, particularly as they are being developed, to ensure that the code is of the best quality and that bugs and general sloppiness of coding are kept to a minimum.

The enterprise gets more useable code first time, with all the savings that are generated by bug fixing, extensive testing and remediation work, and the impact reductions in performance of the application when in production.

The 'bad' part is the way this approach can leave developers with a pretty sour taste in their mouths, as they see themselves not trusted, implicitly incompetent and under perpetual competitive surveillance. The negative impact of this can, if an enterprise is not careful, at least out-weigh the benefits that can be achievable.

Tackling the balance of this `good’ versus 'bad' was the underlying theme of a recent company kick-off day held by CAST Software in Paris. One of the main presenters at the event, Bruce Lee, the Senior VP and Head of Ops and Technology at one of the USA’s big, national mortgage providers, The Federal National Mortgage Association, better known the world over as Fannie Mae, tackled this balance head on.

The business is a good example of making the balance work, not least because while it works with some significant economic numbers it is relatively small as an overall organisation. What is more, because of its high dependence on IT to support its operations, a goodly percentage of those staff are applications developers.

It is, of course, a matter of record that, like many other financial institutions around the world, Fannie Mae was hit hard by the global financial crash and its subsequent bail-out by the US Government. Since then, the business has set about some significant re-development of its important IT services.

This included running five or so waterfall development projects costing some $100 million investment each. Most of this was done by third party contractors and consultants and the business needed a mechanism to ensure that what they were delivering good quality code with no flaws or technical debt problems down the road. This is where CAST got involved, providing waterfall project quality measurement. According to Lee the cost was negligible and they were getting good quality measurement and management results.

The next step was to move to agile development as part of a major digital transformation program and this meant providing justification to get such a move past the CFO. The goal now became finding a way to measure the transformation journey, a process they ended up calling 'Ideas that are hard to measure’. Lee explained:

And they are hard to measure, because software faces an infinite demand curve so measuring the ins and outs of development, the investments and costs, at any one point can be incredibly difficult.

The decision was therefore made to add in the CAST Function Point Module to the existing CAST installation and use it to measure the company’s Agile/DevOps journey.

This is also where the negative side of the distaff came into play, for to start with the developers did not like it. They viewed it as a tool to catch them out, identify their individual weaknesses and, perhaps, put them at risk of losing their jobs. Lee said:

We decided to sit them down and explain that, if we could count function points in waterfall and price them, then count then in agile and price them, we could show the CFO the progress we have made.

There were constraints to get over in getting developers to use the new approach, with the first of them being application on-boarding. When they started that had 10 apps working with CAST and each one took some 40 hours and there are hundreds of apps. So, they decided to automate the process to work with CAST and it now takes 15 minutes of engineering and less than a day of code scanning to do the job. To date they have on-boarded some 300 applications, equating to some 900 million lines of code.

The second constraint was working with developers and compliance. The tactic here was to put the applications scans produced by CAST into the hands of the developers themselves. That way they did not feel they were being judged externally.

The business used to put out 1,000 releases a month. It is now doing 22,000 releases a month to the same environment, and Lee claims the code quality is higher than it was before, which means the run-on remedial costs are much lower:

Agile is often seen as a quick and dirty response to improving software development, but we wanted to use it provide a better and faster response. And that requires engineering.

Developing the developers

Fannie Mae now has so many scans going on – was 100 times a year, now almost 1,000 times a year – that it can now feed the results into an artificial intelligence engine. This has allowed them to move from just examining the quality of the code produced by development teams to predicting which teams might have development difficulties when it comes to releases. This allows the management team to intervene earlier in the process, and has meant that Fannie Mae’s production change metrics have gone up 50% every year for the last three years, whole the incident count goes down 70%.

There has been a degree of inevitable contention with developers in some of this. For example, Fannie Mae is not unusual in having developers that like particular development tools. The trouble is that, while they are easy to set up and work with, they do not always produce enterprise grade or quality output of code. This can be important when that code is part of a multi-year transformation process costing hundreds of millions of dollars. Using CAST is helping them identify which are the most effective tools to use, said Lee:

We are finding that, with CAST, those developers that do the most scans are also producing the most function points, at a higher quality, and at a lower cost.

And the cost issue is important. Comparing function point cost now compared to three years ago, this has saved the company some $200 million over a three-year investment cycle. In this context, CAST has helped provide the detailed financial arguments to present to the Fannie Mae board to show that what they were doing was achieving worthwhile results and a good contribution to the overall plan. It also helped them provide anecdotal information to illustrate the underlying story.

For example, this might be a reference back in time to the effort and costs involved in a task such as adjusting a mortgage interest rate across a number of applications – a job that then took 9 months and cost $1 million and now could be done in a week for $10,000.

It is also good for assessing and maintaining what he calls the Agile Maturity assessment of both the code and the development teams producing it. The current goal is to get all the teams to Maturity Level 3, which is full integration of the CAST toolset and the range of development tools being used. Lee explained:

With an agile mindset within a team the focus is on getting better. Every time they do something you want that team to take some time and do a retrospective analysis….what could we have done better? We use the CAST data as input to their own self-reflection.

We don’t come at it from a compliance oversight point of view, saying 'you’re good' or 'you’re bad'. The traditional ITIL methodologies will show up whether a team is good or bad, and if there is a bad one the CAST data can help identify where improvements are required. So we use it as a solution to problem shown in the ITIL numbers. The teams are left alone with the data to have an honest look and analysis at what they have produced.

My take

There is obvious good sense in enterprises using tools that can help improve the quality and reliability of applications code. Being able to develop and use applications that are cheaper overall, more reliable (and with fewer bugs) and work more effectively is a win/win for both the IT department in the company heroism stakes and for the bottom line of the business itself.

But if it is achieved at the cost of developers feeling they are mere galley slaves under the beady eye of the old time-and-motion man, the damage to corporate reputations could be significant; doubly so if the developers are skilled and experienced.

The Fannie Mae tactic of giving the developers the tools to self-monitor looks a good option, for it gives them the information they need to understand where and how they can get better at their job rather than await the finger of accusation to point in their direction. That is a win for them as well.