Outracing BI avalanches - an agile BI success story

Profile picture for user jreed By Jon Reed July 30, 2014
Can you pull off agile BI in a waterfall organization? Ryan Fenner of MUFG Union Bank says yes - here's how his team pulled it off.

Ever heard that one about getting caught in the BI avalanche? I hadn't either - until last week at the TDWI World Conference - Boston. That's when I heard a presentation from Ryan Fenner, Data Solutions Architect at MUFG Union Bank, about doing agile BI in a waterfall organization. 'We outraced the avalanche,' Fenner joked at one point - the avalanche being the peril of a BI crudstorm where agile gets caught up in the churning gears of waterfall project management.

But Fenner and his team pulled it off. I touched on his story in my show wrap, Big data gotchas and agile BI lessons from TDWI Boston, but there is more to tell. In his presso, formally titled Chasing Unicorns: A Real World Agile Success Story, Fenner laid out how agile BI can work even in a context where the umbrella Enterprise Project Management group is waterfall-oriented.


Shadow IT and agile BI: provoked by market pressures

Fenner developed his agile chops though nine years of experience in 'Shadow IT', or, as he quipped, 'We had servers under the desk in the janitor's closet.' Business-funded Shadow IT was a response to changing market conditions; external factors in the financials industry forced a need for rapid change. Data needs within business groups became more intensive, and business-embedded IT groups blossomed. IT was changing - no, it had to change. Existing IT-business relationships left a lot to be desired. As Fenner put it: 'Business sponsors didn't enjoy funding technology teams.'

Fenner also experienced the frustration of waterfall methods. One huge bugaboo? The 'requirements' gathered at the beginning of those projects didn't translate into a successful result:

We weren't successful with waterfall. I'm not a fan of requirements - the customer thinks they want it, but it's not what they need. You assume the customer knows how to tell you what they want, but you could go 100 different directions with these requirements. A lot gets lost in translation. Then, the final product is not what they wanted. We needed a better way to be effective with limited resources.

A classic slide from Fenner's deck illustrates the requirements gathering problem:

waterfall better

Selling agile to the organization

Nobody wants a waterfall rhino. Another agile driver? Projects could be funded on the fly, with limited red tape. Fenner's team did not have trouble selling the benefits of affordable projects on the fly to the lines of business (who he refers to as 'customers'). But he did have a selling problem: with executive leadership.

Surprise - executives aren't that interested in the agile verbiage of  'scrum masters' and 'sprints'.  And compliance-minded executives certainly don't want to hear anything that whiffs of 'We don't need any documentation anymore' - comprehensive documentation being a typically weak aspect of agile approaches.

So how did Fenner sell agile? By selling to leadership in terms they could support:

agile selling

With buy-in achieved, it was time for the real learning curve: agile in action. Fenner's talk was injected with lessons from two years of agile successes, of the hard-won variety.

Here are some agile BI keys that jumped out:

  • Don't underestimate the importance of testing.
  • Recruit from the business - if you can't get the best talent from the lines of business on board, forget about agile.
  • Don't make the 'horrible mistake' of staffing your agile teams with too many developers - you need process owners/subject matter experts (SMEs).
  • Use application lifecyle management, data warehousing automation, and development automation tools to manage agile cycles.
  • You can kill agile quickly with micro-management - give teams plenty of autonomy.
  • On the flip side, don't let prima donnas poison team environments - your 'scrum master' needs to intervene on behalf of team chemistry.
  • Diverse agile teams with IT and business users work best, but require skillful management.
  • Agile still requires governance - give agile teams guidelines to work under.

Getting the work funded and done

To fund their agile projects, Fenner had to negotiate 'challenging' funding parameters. They ultimately used a Statement of Work across multiple projects, using a charge back model with a business sponsor fronting cash upfront. Merging agile and waterfall from a funding standpoint was another tricky maneuver: the early stages of agile, the planning and design phases, were not easy to capitalize.

The development phase that mixed in with the iterative design and development stages could be capitalized. Over time, Fenner figured out how to translate agile sprints into buckets that waterfall-oriented managers could understand and approve. Sprints were referred to as 'design'. Dev/test/deploy - 'POC'. Red tape cleared - the work went on.

As they got deliverables out the door, Fenner's team settled on two week sprints. One week was too short, a month was too long. And, unlike many agile projects, they were able to incorporate remote team members via webcams without issues (Involving remote team members is usually one of the toughest parts of agile projects).

One interesting aspect? There simply wasn't time to ensure complete data accuracy. They did measure the accuracy of the data and give that info to the stakeholders. Instead, Fenner's team emphasized data integrity as a goal - if there was an inaccurate piece of data - or an accurate record for that matter - it would appear that way in all views. For their agile architecture, they allowed a scenario for data quality to be addressed further down the road, to allow for accuracy to catch up with integrity.

Given these tradeoffs, why was agile successful? Well, for starters, how about early tangible delivery, as in: 'an actual datamart business could use!' Business users were highly engaged by the agile teams. New systems and groups could be quickly onboarded. Best of all: continual feedback loops led to a better product. No more rhinos. And the IT leadership team was happy with the improved business relationship - and more robust data warehouse.

Final takeaways

Fenner left the group with a few more lessons: turn your SME to developer ratio upside down. Source your scrum master and the product master from the business and bring them into the IT team. Communication skills and intellectual property are critical issues to building strong agile teams. Use a Center of Excellence (COE) model to align orgs and operating models.

Final tips:

  • Drop the scrum books - no one does it right, you don't have to either. Pursue the value of agile, not the process itself. Take some liberties with it to make it work for your group.
  • Don't let perfect be the enemy of the good. If it's 80 percent good, move it forward.
  • Recognize technology debt - add to your backlog when you cut corners. Establish scrums to fix hacks and remove legacy burdens.

Fenner cautioned the customers in the audience that agile is not for everyone, or every situation. He warned the audience to expect resistance if they go the agile route - it's not just a different management approach, it's a cultural shift.

To drive his point home, Fenner referenced the Tom Hanks film 'Road to Perdition'.  Fenner: 'Agile is a road of eternal punishment and purgatory. We've been doing it since 2008, and we are still fighting. It takes at least two years to get good at this.'

Easy? No. But for those who think agile BI can't be done in a waterfall context, Fenner has some news for you: it can.

Image credits: images and slides provided with express permission of Ryan Fenner. Feature image credit: Woman Snow Skier on a dangerous, steep slope © Alex - Fotolia.com.

Disclosure: TDWI provided me with a press pass to TDWI Boston. Fenner presented this content as part of the TDWI Executive Summit BI track, which I attended. Diginomica has no financial ties to TDWI or MUFG Union Bank.