Agile project delivery has now overtaken traditional project management as the preferred method for delivering software projects. As the use of agile delivery continues to expand, it is worth considering why this method has become so popular. Traditional project management, or Waterfall, consisting of a sequence of analysis, planning, and development towards a major deliverable, has been the dominant methodology for delivering value for nearly 50 years. Winston Royce first described a phased approach to technical problem solving in a 1970 paper, and this laid the foundations for the Waterfall approach. The PMI was formed in 1969 and has been promoting waterfall delivery for much of its lifetime. And for good reason. Waterfall has been responsible for successfully delivering value for decades.
But we need to consider why the mantle for successful value delivery is shifting from Waterfall to Agile. Is it simply that one framework has become the preferred approach, or is there something fundamentally different between the methodologies? The answer is tied to the problems we are solving. While both traditional Waterfall and agile delivery are used to deliver projects or solve problems, they approach things in a different way.
Scott E. Page is a professor of complexity, social science, and management at the University of Michigan who categorizes problems in terms of landscapes. Simple problems are like smooth, symmetrical landscapes. Think Mount Fuji. Every step we take towards the summit is a step towards the solution (in this case, getting to the summit of the mountain is solving the problem or delivering value). In more adventurous mountain ranges, we must think a little outside the box. While every step uphill does not always take us to the summit, we can start imagining how to get to the summit and plan a way to get there. These simple or adventurous landscapes are ideal for an approach like Waterfall, which can be planned out with a high degree of certainty from the start. These are the domains in which traditional project management practices excel. We can take our time to understand the landscape and plan out our strategy to reach the summit. We can create a work breakdown structure to plan the ascent of the mountain in as much detail as necessary, we can draw up the plan and track our progress toward the summit.
However, Professor Page describes other kinds of problems – complex problems – as dancing landscapes. The land shifts continually, more like an active volcano with many fissures opening up and continually flowing lava fields. The volcano is a dancing landscape. Reaching the summit is a very different proposition. Taking time to understand the landscape quickly reaches a point of diminishing returns. Planning a fixed route to the summit doesn’t work. In dancing landscapes, a successful strategy for reaching the summit is based on continual discovery and change, of taking small steps forward followed by realignment to the goal. A plan of action is necessary – small steps forward to explore the landscape. Any plan is temporary and acts more as a guide than a firm commitment we can follow with confidence. This is the world of agile delivery.
If we recognize that the category of problem we are trying to solve leads us to a preferred value delivery framework – Waterfall for simple or adventurous landscapes and Agile for dancing landscapes – we can begin to understand the trend towards agile delivery in technology. In the past 20 years, driven in part by a shift to a world that is networked and internet-enabled, we have seen a migration from simple or adventurous landscape problems to problems more closely aligned with dancing landscapes. The problems we are solving are more likely to be complex.
As a leader we want to start by understanding the problems we are solving, the landscapes we are working in. We want to choose our problem-solving approach based on the category of problem we are solving. Furthermore, if we recognize that more and more of our problems are complex problems, we also need to increase our capacity to solve complex problems. This is not simply more agile teams; it requires the right guidance and governance. Waterfall projects and agile projects require a different approach to planning and behave differently as they progress. It means upgrading the institutions we have relied on for decades to guide our problem solving (project delivery). The PMO (Project Management Office) has been the centre of excellence for Waterfall delivery for decades. A shift to more agile delivery requires a different governance approach that is incorporated into the PMO. This new institution is adept at guiding and governing in dancing landscapes.
Once we realize that the difference between Waterfall and Agile is that each is optimized to solve different categories of problems, we can begin to understand the shift from Waterfall to Agile in technical project delivery. We might also anticipate that more and more problems, technical or non-technical, will present as complex problems more suited to agile approaches than a traditional waterfall approach.
