When Is the Risk of New Methods Worth the Reward?
Vodafone combined agile development, data warehouse automation, and flexible data modeling to transform its reporting-centric BI into next-gen analytics.
- By Steve Swoyer
- November 1, 2016
You shouldn't be afraid of failing. Instead, you should expect to fail.
That's what Jos Driessen, manager of business intelligence strategy and architecture with telecom giant Vodafone Group LLC, told attendees during his session at Teradata's 2016 Partners Conference. Driessen was speaking about Vodafone's ambitious effort to build and manage what he claims is the largest analytics data warehouse in the telecom space.
Vodafone's project wasn't a failure -- Driessen was promoting it at a major industry conference, after all -- but it was risky. Any analytics development often produces unusable or inconclusive results.
As analytics become more critical to business strategy, growth, and success, failure will become more common. By traditional metrics, analytics development is risky, Driessen conceded -- but the risk is worth the reward.
Putting Information in the Driver's Seat
Driessen's idea was to complement Vodafone's use of a best-of-breed ETL product (Ab Initio) with data warehouse automation software (WhereScape RED).
Vodafone would use both technologies to feed its Teradata data warehouse. For good measure, Driessen tacked on an additional complication. He wanted to transition away from the Teradata Logical Data Model Vodafone had been using and instead implement Dan Linstedt's Data Vault 2.0 model.
Driessen liked Data Vault's ability to provide highly granular historical data -- and he also knew Data Vault could be used with agile data warehouse development methodologies.
Add it all up and Driessen had a plan for massive change. Because massive change entails massive risk, not all companies (or architects, for that matter) would have undertaken such a project. Driessen didn't view what he was doing as just another "project," however. In his fail-fast philosophy, "project" is a dirty word. He prefers the word "initiative."
Getting Comfortable with Agile and Failing Fast
The distinction isn't merely semantic, he says. "We don't call them 'projects' anymore. We call them 'initiatives.' We don't have project managers, we have agile fail-fast coaches."
The distinction matters because agile development, in which a project or initiative accretes in a series of time-limited sprints, is a completely different way of doing software development. It requires different thinking. This means jettisoning the terminology of traditional, waterfall-based software development.
It also means getting a lot more comfortable with failure, Driessen says. He told the following story to illustrate his point.
"I hired someone from IBM Global Services, someone new [to BI and data warehousing]. He was building this data mart and even though he worked on it and worked on it, it just wasn't a success. He came back to me and he didn't know what to say. 'I don't like to fail,' he told me. 'You didn't fail,' I told him. 'You learned what doesn't work. You learned why it doesn't work. Now you can start over.'"
Agile Methods Require Agile Tooling
Starting over is often cost prohibitive -- particularly in waterfall-based software development, where requirements are codified up front. It's different in an agile-based approach.
Because agile breaks a project -- or "initiative," in Driessen's terminology -- into iterative deliverables, it's much more tolerant of missteps, miscalculations, or near-complete restarts.
An agile methodology must, however, be complemented with an agile software development tool, such as a data warehouse automation (DWA) tool. Vendors including Attunity, Magnitude Software, TimeXtender, and WhereScape market agile-oriented data warehouse automation tools. One of the selling points of DWA software is that it provides a degree of protection -- a kind of safety net -- against failure. With most data warehouse automation tools, you can back out of a blind alley -- a "failure" -- and begin with your last known good milestone.
This was critically important to Driessen. When he first came to Vodafone, he was given a mandate to transform Vodafone's reporting-centric business intelligence (BI) practice into a next-gen analytics practice: to "put BI in the driver's seat for new business," as he puts it.
Analytics development is inherently risky. It requires analysts, data scientists, and developers to experiment -- to hypothesize, test, and analyze -- as they build. Agile data warehouse development combines this element with still another variable: ongoing collaboration with the line of business to design and build operational applications.
You Are Going to Fail, So Make the Most of It
Dreissen fully expects the business to be dissatisfied with the first software his developers and architects produce. They're probably going to be dissatisfied with the second revision, too.
This isn't just okay, he says: it's necessary. It's part of the process. You can't and shouldn't eliminate it. If anything, you should try to learn as much as you can from it. Make the most of failure, he says.
The key to agile data warehouse development is to get it right -- eventually. The faster you can fail -- i.e., figure out what doesn't work -- the faster you can try something else.
"If you fail, you can revert back ... [and] let WhereScape [RED] regenerate the Data Vault model. You can [incorporate] what you learned about what didn't work," he argues. "We always know that if we totally screw up and the customers don't like the end result, we can revert right back [to where we started] and in a couple of hours, we [can] generate a totally different [Data Vault] model.
Vodafone's Bet Pays Off
Two years on, Vodafone's big risk is a rip-roaring success, Driessen points out.
"I can now present the customers with data that's one hour old. We've accelerated the delivery [of applications]. The time it takes to add or change things [that are missing, not quite what the business wanted, etc.] to an application has gone from months to days," he says.
"We've saved [money], too. We're not paying as much for Ab Initio because we don't use it as much. With what we saved [in operating expenses] in the first year, we were more than able to pay for [our] WhereScape [licenses]."
About the Author
Stephen Swoyer is a technology writer with 20 years of experience. His writing has focused on business intelligence, data warehousing, and analytics for almost 15 years. Swoyer has an abiding interest in tech, but he’s particularly intrigued by the thorny people and process problems technology vendors never, ever want to talk about. You can contact him at evets@alwaysbedisrupting.com.