Why the Waterfall or Agile debate will be around forever
Which is the right methodology to use for your project: Waterfall or Agile? The industry may be at peak Agile, as the practicality of Waterfall is winning over more converts.
After all these years of the Waterfall or Agile debate, there's no doubt that both have their pros and cons. So, it's surprising there are still evangelists on each side dead set against finding middle ground. For most technologists who have been involved in waterfall or Agile projects, it's possible to argue either side of the debate. In fact, there's rarely a project or organization that uses only one framework these days. There are some aspects of Agile that get roped into Waterfall and some essential disciplines of Waterfall that just can't be cut from even the purest Agile project. Here's a look at why software development will always feature a bit of both worlds.
Choose a Waterfall or Agile approach
A team has to balance many factors to determine the right methodology for a project. Agile tends to value:
individuals and interactions over processes and tools;
customer collaboration over contract negotiation; and
responding to change over following a plan.
Product-focused project management proponent Curt Raschke pointed out this is by no means a complete condemnation of Waterfall.
"If you look at the original Agile Manifesto, it never says the other attributes are wrong," Raschke said. "It just values the Agile side more."
Don't assume these Agile-friendly factors should always be given the most weight.
Balancing Waterfall and Agile
In Raschke's view, on any given project, a team might need the control and centralization of Waterfall and the flexibility of Agile. One example is software in the medical industry. In a Food and Drug Administration regulated area, it's necessary to have certainty and control. For other fields with limited oversight, teams can use Agile and innovate.
"What's wrong is to assume one or the other is always the answer," Raschke said. "When you look at a product and it needs flexibility [or] the client is not sure what they want, what the market needs or there is innovation required, Scrum is important. If it is embedded or mission-critical software, Waterfall might be a better choice."
What phase the project is on also determines which methodology a team should use. Business case development, choice of infrastructure and decisions on deployment and maintenance are all aspects of the software development lifecycle that don't necessarily lend themselves to the Agile approach. When forced to align with either Waterfall or Agile, these business cases tend to fall more in line with Waterfall. Once those outlines are in place, the actual development might trend more toward either Waterfall or Agile, depending on the purpose and nature of the project.
Raschke said to remember what is most important. "The question is: What do we need to do to get it to the customer? In that case, it usually ends up as hybrid."
Waterfall or Agile for responsiveness
Jay Packlick, an enterprise Agile, thinks Agile offers more adaptability in a world of rapid change. For him, planning is not something that is done just once at the start of a project. It is an ever-evolving process.
To decide between Waterfall or Agile planning, Packlick asked a vital question: "Are you going to do most of the planning in setting the direction or using information available to steer toward your direction? In a world of complexity, what's critically important is that we update our information with what is true now and we change our actions to meet our objectives."
When organizations go Agile and give responsiveness a high value, they'll be faster to market, with greater innovation and more significant ROI. There are some rewards, however, for organizations that follow a well-crafted plan and stay on a strict schedule. Mark Yarbro, performance test manager at Bank of America, pointed out that flexibility isn't always the most crucial aspect of a project. Sometimes, structure rules the day.
"We have 1,200 applications," he said. "We release many of them on a 90-day cycle. The way it works is we get everyone together so they are working in lockstep."
The Waterfall and Agile hybrid
A disciplined Six Sigma approach with cyclic Waterfall has proven the smart move overall for Bank of America. There are a few Agile teams within the company, but they have to stay on schedule with the rest of the rollouts. It's simply not possible to allow the type of free rein most Agile shops thrive on. As Yarbro pointed out, if people can't load a page on a shopping website, they just come back later (or not). If they can't get into their bank account, they get on the phone to find out why -- any major website failure in banking would end up as front page news.
It's not just large organizations that recognize the benefits of structure for far-reaching projects. Business development and project management consultant Devin Ellis said he has seen this type of mixture in action in the faster-paced services world as well.
"My last assignment at a consulting firm was a hybrid approach," he explained. "Some projects were fairly strictly Agile. But if it tied into larger initiatives, they did have gates set up that were more like Waterfall."
While it is essential to deliver a workable product, too much focus on functionality and not enough focus on nonfunctional requirements can end up being unsustainable. Yarbro has seen this issue with Scrum when the rush of getting things accomplished rapidly becomes an excuse to forego documentation. For a startup looking to build and sell a product quickly with no aftermath, that approach might work. But an enterprise has many more issues to consider.
"The actual development is the tip of the iceberg," Yarbro said. "The maintenance is what's under the water."
When the developers who originally worked on a project leave or retire, the organization can spend significant resources to figure everything out all over again the next time it has to revamp or update existing software. Accountability in documentation is essential.
Agile or Waterfall for organizational culture
In the end, human nature guarantees that the old ways of doing things stick around. Even when it could transform into something better, not every organization or team can shift from Waterfall to Agile.
Packlick said this is a reality he encounters as a coach.
"When you have an awareness of the values of the system, you understand what's possible. There are command-driven cultures, outcome- or results-driven cultures and, rarely, there are cultures that are focused on growing people. Change must match the culture."
But not every culture can handle that type of transition.
"You must determine if the culture is healthy enough to engage in change," he said.
Everyone has to say they are an Agile shop to attract new talent.
Mark YarbroPerformance test manager, Bank of America
Agile is based on the assumption that people will, more often than not, work well together and make good choices. Packlick pointed out that a team at the high end of the dysfunction scale must shift how they operate as human beings if they are to embrace a highly collaborative framework, like Agile.
In fact, Agile is not quite as prevalent as it might appear. Just because a company says it is Agile does not necessarily mean it is, Yarbro explained. Often, it's really about marketing a particular workplace ideal.
"Everyone has to say they are an Agile shop to attract new talent."
That doesn't mean the entire culture has to be one way or the other. In a typical organization, there may indeed be teams that can be handpicked to go Agile for certain projects, while lower-skilled, short-term workers and those who require more direction can be led through traditional Waterfall workflows.
Look to the future
Waterfall will definitely stick around for those organizations that place a high value on structure, stability and well-defined scope. However, a hybrid or responsible Agile approach is likely to become the norm as software development continues to mature. This might look like Agile with better documentation built in, QA that is more deeply involved in Scrum and teams with a stronger commitment to understanding how decisions made on the fly will impact maintainability over time.
Dig Deeper on Software development best practices and processes
