TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

From Waterfall to Agile: Three Lessons Learned

Learn from real-life experiences of switching from waterfall-style development to completely agile analytics.

• By Shikha Verma
• March 21, 2016

Data and analytics projects have traditionally been performed using a waterfall methodology, creating a bad reputation for multi-month delivery cycles and slipped timelines. Although much has been written about the need for agile and the theoretical way to run agile projects, there is hardly any help available for realistically moving from a waterfall methodology to a completely agile one. Here's my real-world experience of my team's transition.

In waterfall projects, the requirements-gathering phase ends in requirements being signed-off and handed over to the design team. The design team architects and designs the solution and then hands off their work to the development team, which creates the solution. The solution is turned over to the testing team, which tests the solution and, after several back-and-forth exchanges with the design and development teams, corrects errors and verifies the final solution against the set of requirements. The solution is passed back to the requirements team for user acceptance testing and validation. Design, development, and testing issues are resolved after the three teams make corrections and re-verify the results.

How cumbersome and bureaucratic did that paragraph sound? Extremely, I would say. Although this approach works for many complex projects, it involves many sign-offs and back-and-forth blame games and considerable rework.

The origin of this project management methodology came from the construction industry where high precision was a necessity because lives were on stake, which justified the approach although projects took months or even years.

Analytics projects are very different. The waterfall methodology is not suitable. Analytics projects lend themselves much better to an agile style of project management, in part because the product being developed is very visual. The best way to describe an agile project is to make bite-sized chunks -- and chew, chew, chew! That's how a foodie like me always describes one of agile's key characteristics, and the analogy resonates with most people.

To make the transition from waterfall to agile for your analytics projects, there are three key lessons I learned from my own experiences. I learned these through taking my own teams through this tough transition in a few different companies.

Lesson #1: Create a unified team

This is unarguably the most important step you can take. Create a team that includes people from the all parts of the project's life cycle. For example, if you are creating a marketing analytics tool, include the marketers who will use the tool in their decision making, the data experts who will gather and organize the data, and the tech team that will build the tool and test it.

Very often, I see teams and companies getting trapped in their organizational barriers. The whole point of this step is to knock down those barriers and create a holistic, unified team that gels well and can make swift decisions.

Lesson #2: Fail fast, design for flexibility

I've found this step to be the hardest for the technology teams to internalize. Being detail-oriented and very analytical by nature, it is quite hard for seasoned technical professionals to adopt the philosophy of "fail fast." They often get trapped in wanting the use cases and requirements to be written down in extreme detail so they don't make a mistake in designing and developing the technical solutions.

As a leader of such teams, your most critical job is to enable the team to go fast, accept mistakes, and learn from them. This is even more critical in analytics projects because nobody really knows what analytics will make sense until they start using the tools and see the possibilities. Analytics tools are visual in nature. Depending on the data and usage, the usefulness of each visualization changes rapidly. Your team needs to experiment with many views and keep the solution flexible.

Lesson #3: Prepare for rework

If you take Lesson 2 to heart, you will have rework. I have found that the rework can be quite drastic in the first three or four iterations of agile analytics projects before the solution starts to solidify. If you find yourself re-designing your data model in iteration 2, you're doing great!

Take the fear of rework off, instead prepare for it. Keep the design principles simple and flexible – that would make rework less painful. The key aspect is to set expectations across your team and your stakeholders to not expect perfection… rather to adopt an experimental mindset.

Applying the Lessons

Keep these key aspects in mind and you are sure to make strides as you move from waterfall to agile. Pair that up with the right strategy and tools, and you will be on your way to money-making analytics.