Big data gotchas and agile BI lessons from TDWI Boston

Jon Reed Profile picture for user jreed July 24, 2014
I'm fresh off two days on the show floor at TDWI's World Conference - Boston, where all things BI, analytics, big data were hashed. Here's a few highlights from my notes - some of which came as a real surprise.

Boylston dinner
I'm fresh off two days on the show floor at TDWI's World Conference - Boston, where all things BI, analytics, and yes, big data were hashed (the show runs through Friday, track the live chat here - scenic Boston summer to your right).

There were plenty of meaty case studies (more on those in future pieces), and as usual, some of the best conversations happened in the exhibit hall where sponsors pitched their wares and customers grabbed a coffee moment. Here's a few highlights from my notes - some of which came as a real surprise.

1. Cloud BI is still in limbo between genuine trend and marketing exercise. I was expecting a bit more gusto behind cloud BI solutions, but from a customer angle, it just didn't connect in a jugular 'we need this, and we need it right now' way. During a cloud-focused BI presentation Wednesday, David Linthicum noted trends such as a 20 percent increase in cloud-hosted data analytics from 2013 to 2014.

He projects cloud BI usage by customers for 2016 and 2017 to be in the 90 percent range. Most of these scenarios involve hybrid, or, if you like, 'multi-cloud', which add complexity to the appealing simplicity of cloud deployments frighteningly fast. The attendees I spoke with seemed to accept the cloud transition of IT as an inevitability, but that doesn't mean it's today's priority. However, Linthicum made one point about cloud BI that hit home: 'cost and time to market drive enterprises to the cloud for BI projects.' Which leads us to...

2. Agile BI is becoming a credible response to business user frustration. I hesitate to even use the world agile because of all the assumptions it invokes. But several customer case studies and presentations returned to the agile theme. A dual presentation from GE Aviation's Timothy Giesk and Mohammad Alagha featured the on stage back-and-forth between an IT dude (Giesk) and a business type (Alagha). Giesk presented an all-too-familiar IT scenario of business user frustration with clunky data warehousing projects. GE Aviation's goal? 'Reduce the cycle time from data to decision.' Easy? Nope. Necessary? Well, as Alagha put it, 'This was IT's last chance.' As in: one last chance to get the business-IT relationship right.

Timothy Giesk

So, Giesk's team made changes. As he put it: 'I embraced the risk.' What he found? An agile process allows for a much faster, immediate result, which brought out necessary feedback from business users sooner. Involving the business more deeply is not as simple as it sounds, but as Giesk and Alagha put it, business, not IT, understands the data. Therefore, business should drive functional strategy and own data quality. The point, then, isn't about agile buzzwords but a faster time to BI value with a more collaborative approach that (hopefully) avoids the dreaded 'This wasn't what I wanted' meeting. Business requirements change fast, and may not get properly clarified until a product iteration can be tested.

Another presenter, Ryan Fenner of MUFG Union Bank, in a memorable talk entitled 'Chasing Unicorns: A Real-World Business Success Story Using Agile BI Practices in a Complex and Changing Environment,' warned that if you don't involve your best people - including your best business users/subject matter experts, you can forget about a successful 'agile' outcome. Fenner still has to function inside a waterfall-oriented organization - another example of the obstacles data innovators have to overcome.


3. Enterprise data warehouses aren't dead, but they are facing massive challenges
. Remember that fantasized time when all the data the enterprise needed to make strategic decisions would live in one happy, clean place - the Enterprise Data Warehouse? Well, if that time ever existed, it's gone now. For compliance and governance reasons, the EDW still has a role to play, but new external data sources and web-based systems are pushing enterprises to 'let data live where it lives' - at least in the short term.

This is where cloud BI and/or agile trends fit in also - spinning up quick solutions to prove value. I heard a fair amount of 'minimum viable product' talk - working outside the EDW is often the best way to get analytics tools out fast enough for the business to use IT's solution - and not their own.

Yes, the EDW might stick around, not unlike the ERP transactional system of record, but pressing needs dictate lightweight approaches, without the burden of conventional data transformation. 'Latency' is becoming a dreaded word in this context. Waiting for IT to create pre-defined reports is losing viability, and fast. Executive Summit Presenter James Taylor illustrated the ludicrous results of data-delayed decisions in modern user cases like fraud prevention:


4. Big data is more than a curiosity, but buyers are still kicking tires. The vendors at the show (which ranged in size from upstart BI players to IBM) were more aggressive about the big data conversation than the customers I spoke with. Example: I had lunch with an automotive customer (IT manager, reporting into the CIO) who has investigated Hadoop. But for now, they have a smaller SQL database that serves up the analytics the front line teams need. 'In our organization, we respond to pain first and foremost, and we don't have big data pain yet,' was how she put it.

When I asked her if there might be opportunities in that data, not just pain, she said definitely - but, her team doesn't have the time to delve into data discovery. They are too busy serving the analytics needs of the business and maintaining existing systems. The big data sessions I looked in on were well attended, with customers asking questions about the proliferation of big data vendors and architecture. During a full day session on big data analytics, presenter Krish Krishnan hit on another major big data challenge: skills shortages:

Krishnan was more optimistic about recruiting R programming skills, with 25-28 year old programmers being a sweet spot. He also cited a number of organizations, including Cisco and Netflix, that have been successful upskilling their own data or business analysts into data scientist type roles. But big data skill issues must be added to the mix of factors that are causing big data projects to proceed cautiously.

Final quick hits from the show

A few more quick takes:

  • Self-service BI, or the democratization of BI if you prefer a loftier term, was a key project driver. One example was Elizabeth Gray, presenting on behalf of the Austin Fire Department, where BI apps and Qlikview dashboards have driven user adoption and led to significant cost savings, e.g. 4,893 hours of data compilation time saved via BI reporting. Even more important: better citizen response times via locational resource pooling and so on.
  • 'Mobile BI' was not flagged as a high priority topic. ('We're just trying to get our operational apps on our mobile devices, forget about BI' - one attendee said to me).
  • Like big data, 'real-time' is a growing factor. But what real-time means, and what it costs, is still open for more debate than some BI vendors might like. But - there is a growing sense that latency in data processing is unacceptable for the majority of data projects. Whether that means in-memory processing or frequently updated appliances depends on the use case.
  • During the wrap of the Executive Summit program, which ended Wednesday, TDWI's Fern Halper and David Stodder hit on the importance of tying analytics back to the data warehouse, and jump starting advanced analytics projects by connecting to the performance metrics already being measured. Insights from historical data prompt a review of the decision model, and, ideally, predictive scenarios. And: the missing 'V' in the big data equation (Volume, Velocity, and Variety) is still Value.

There was plenty of meat on the bone at this show. Customers had enough ROI examples to drive further investigation of BI solutions, including the latest big data, real-time, and NoSQL options. But, how those solutions will be deployed, which databases will get the workloads, and how IT and business can get on the same page enough to produce a positive outcome - that's a work in progress.

Image credits: Boylston street (outside the Hynes convention center) and presenter Timothy Giesk - by Jon Reed. James Taylor fraud slide used by express permission of James Taylor.

Disclosure: TDWI provided me with press access to TDWI World Conference Boston.


A grey colored placeholder image