The Agile Fallacy: Why the Dominant Development Methodology Needs a New Operational Standard
For the past two decades, the technology industry has been dominated by Agile development practices. Agile is a methodology that emphasizes iterative development, flexibility, and collaboration among cross-functional teams. The approach has become the go-to standard for software development, and many organizations have implemented it with the aim of becoming more efficient and responsive to changing market conditions.
However, 30 years ago, the industry attempted to import Lean practices, which was a methodology based on the Toyota Production System (TPS) in the 1960s. Lean practices aim to minimize waste and maximize efficiency by optimizing the value stream. But unfortunately, the technology industry’s attempt to import Lean practices failed. Instead of “continuous improvement,” progress halted. This failure led to the rise of Agile, which has since become the industry standard.
Despite its widespread adoption, Agile has fundamental flaws that have been apparent for some time. The first flaw is that humans are not machinery. Agile’s emphasis on efficiency and productivity can sometimes lead to teams being treated like cogs in a machine rather than individuals with unique strengths, weaknesses, and working styles. This can lead to burnout and a lack of work-life balance, which ultimately hinders productivity.
The second flaw is that design is not inventory. Agile development methodology tends to focus on producing a lot of code quickly.
The third flaw is that product cannot be defined by what can be accomplished by an arbitrary number of people of arbitrary skill and experience in a two-week sprint. Agile’s focus on sprints can lead to a narrow focus on short-term goals rather than long-term strategic planning. This can lead to a lack of vision and a failure to develop products that meet the needs of the market in the long-term.
The origins
To understand how we got to where we are today, we need to look back at the origins of Agile. In 2001, a group of software developers met to discuss ways to improve software development processes. The result was the Agile Manifesto, a set of guiding principles for software development that emphasizes collaboration, flexibility, and responsiveness to change.
Before Agile, the dominant development methodology was Scrum, which was developed in 1995. Scrum is a subset of Agile and focuses on team collaboration and iterative development. Scrum has been successful in many organizations, but it too has its limitations.
As startups refocus on “operational efficiency,” it’s essential to emphasize that efficiency is a way of working, not just headcount. The focus should be on creating an efficient working environment that enables teams to work collaboratively and produce high-quality products that meet the needs of the end-user.
Elimination of waste
The most thoroughly documented aspect of TPS was the elimination of waste, which came in eight different types. The most detrimental types of waste were excess inventory, both finished products and raw materials, and idle machinery. Toyota’s solutions to these issues were logistical in nature and came to be known as “just-in-time manufacturing.” This methodology involved receiving materials just in time for the assembly line to process them, just in time for the finished product to meet customer demand.
In addition to TPS, Toyota also had a philosophy called “The Toyota Way.” This philosophy called for continuous improvement, including overcoming challenges towards a long-term vision, kaizen, and genchi genbutsu. Genchi genbutsu is a principle that means “real place, real thing.” It encourages decision-makers to go and see for themselves before making decisions, and it also encourages a bottom-up approach to improvement, gathering insight from every employee, no matter their level.
In the United States, TPS was rebranded as Lean in a 1988 article titled “Triumph of the Lean Production System.” Lean and its forebears are perfect for the manufacture of physical goods, particularly when there is an established market and a mature product. It relies on the predictability of both demand and the supply chain.
When it comes to creating a new product, manufacturers rely on different processes in their research and development (R&D) departments. Whether it is Genentech developing synthetic insulin or Proctor & Gamble developing the Swiffer, the process is long and linear. These involve deep research and market analysis ahead of product development.
Agile methodology
The burgeoning software industry largely followed the Waterfall methodology in its early years. Waterfall methodology is a linear development process that culminates in a fully realized product release. However, a legitimate criticism of Waterfall is that the big, expensive product release can still be a dud. As such, Waterfall is used as a foil to argue that iteration, testing, and reaction to the market are necessary on a shorter timeline.
In the 1990s, practitioners in the technology industry explored ways to apply Lean practices to digital products. However, they misdiagnosed the problem from the beginning. Time to market is important, but it’s more so from a cash flow perspective, particularly for startups. Lean proponents threw out the baby with the bathwater. Note how Scrum and Agile literature make little to no mention of research or strategy. Design got a bad rap as the lengthy, time-wasting phase of Waterfall. Lean frameworks are all about build and release.
Around this time, there was a “Kaiju vs. Mecha” style fight between proponents of Lean and those who had been practicing Waterfall development. Both the monster and the robot failed to consider users’ needs and instead stomped on them and destroyed the world.
The practice of iterating on a living product in the market is a luxury of software. Updates can be pushed at any time, and assembly lines and tooling don’t need to be revamped. Every technology product should be built in stages, but it also must be built towards an end goal and a fully realized product. Success requires strategy, research, design, and genchi genbutsu, just as The Toyota Way intended. The Agile vs. Waterfall argument is a false dichotomy: a company can have a long-term strategy and carry it out while still releasing the product incrementally, iteratively.
Criticisms of Agile
One of the primary criticisms of Agile is its incompatibility with research, design, and scalable development. Agile’s focus on sprints can lead to a narrow focus on short-term goals rather than long-term strategic planning. This can lead to a lack of vision and a failure to develop products that meet the needs of the market in the long-term. Moreover, the Agile methodology does not place enough emphasis on user experience and design, which are critical to developing software that is user-friendly and meets the needs of the end-user.
To overcome these limitations, it’s essential to adopt a more comprehensive approach to software development. The approach should be based on a combination of Lean, Waterfall, and Agile methodologies, with a focus on long-term strategic planning, user experience and design, and scalable development. This approach should also incorporate genchi genbutsu, encouraging a bottom-up approach to improvement and gathering insight from every employee, no matter their level.
Scrum
Scrum has several components that make it the way it is. It is timeboxed into a Sprint to ensure that software can be released into the market in an iterative cycle. Sprint planning occurs just before the sprint, when the latest insights on what to focus on are available. There is a daily standup to allow for on-the-fly operational improvements, but only developers are allowed to speak, and discussion is not allowed because it leaves the machinery idle. At the end of the sprint, there is a review to view working software and determine what was not completed or what needs improvement. There is also supposed to be a retrospective to discuss what went well and what didn’t go well, in the interest of increasing the quantity of output. The backlog isn’t core to the Scrum practice; it’s an artifact of the collateral damage and detritus of everything jettisoned during the sprint.
However, Scrum has some illogical components. Scrum’s emphasis on sprints can lead to a narrow focus on short-term goals rather than long-term strategic planning. Moreover, Scrum’s focus on efficiency and productivity can sometimes lead to teams being treated like cogs in a machine rather than individuals with unique strengths, weaknesses, and working styles. This can lead to burnout and a lack of work-life balance, which ultimately hinders productivity.
A comprehensive approach
The Agile manifesto and Scrum practices have devolved into something caustic for organizations. They were well-intentioned at their inception, aiming to treat developers as humans rather than cogs in a machine. However, these principles have devolved into something that allows scrum pods to absolve themselves of any accountability. The combination of Agile principles and Scrum practices is disastrous for startups. These are operational directives from management, and designers, PMs, and engineers are not self-organizing and choosing to work this way.
Research and design create solutions to customer problems, but what gets built always pales in comparison. Product managers are rushed to create the next shiny object on the roadmap with as little effort as possible. Engineers are treated like machinery on an assembly line, always expected to produce and deliver, incentivized to cut corners to get it done.
Leadership is left scratching their heads, wondering why the product is so wonky and ineffective. Far from eliminating waste, Agile and Scrum create only waste. It’s a muddy slop of burnout, tech debt, design debt, a ballooning backlog, hard-coded front-end logic, and an ever-present threat of a complete refactoring.
To get out of this cycle, we need to adopt a more comprehensive approach to software development that takes into account long-term strategic planning, user experience and design, and scalable development. We need to emphasize the importance of research and strategy in the product development process, and we need to allow for iteration, testing, and reaction to the market on a shorter timeline.
It’s essential to adopt a bottom-up approach to improvement and gather insight from every employee, no matter their level. We should also embrace the concept of genchi genbutsu, meaning “real place, real thing.” This principle encourages a bottom-up approach to improvement, where leaders go and see for themselves and make decisions accordingly.
In addition, we need to shift our focus away from time to market and short-term goals and towards a long-term vision. This will allow us to develop products that are truly innovative, rather than simply reacting to market trends. We should also encourage a culture of continuous improvement and innovation, where employees are empowered to suggest new ideas and approaches to the development process.
In conclusion, Agile and Scrum practices are no longer sufficient for the software development industry. To create truly innovative products and avoid waste, we need to adopt a more comprehensive approach that emphasizes long-term strategic planning, user experience and design, and scalable development. We need to embrace a bottom-up approach to improvement and encourage a culture of continuous improvement and innovation. By doing so, we can create products that truly meet the needs of our customers and drive long-term success for our organizations.
