You're too quick to dismiss Agile

I see too many programmers today thinking that Agile is nonsense. It has even become an opportunity to sell books on esoteric methodologies.

However, there's widespread confusion:

• about what Agile actually proposes,
• about how bad the alternative is,
• about what our personal experience really is – usually with Scrum.

First of all, let's quickly remember what Agile really is: a reaction to Waterfall.

Waterfall

Agile was the alternative way of building software discovered when everyone was using the obsolete Waterfall method, and failing hard.

Waterfall means each development phase is done for the entire scope of the project and its output informs or cascades into the next phase:

Requirement gathering & analysis ↘
   System architecture and design ↘
                    Implementation ↘
                            Testing ↘
               Delivery / deployment ↘
                            Maintenance

Waterfall had many serious problems:

• It naively assumed that the system could be planned in its entirety; it did not foresee changes in scope.
• Each step took months to do, so projects took year(s) to complete.
• Clients did not participate in the development process, they only saw the finished project at the delivery phase.
• For lack of communication, invariably, clients felt what was delivered had almost nothing to do with what they had said they wanted.
• By the time the project was completed, the plans were obsolete: so much time had passed that the solution was born inadequate. Even if there had been no communication issue.
• Each phase was driven by fear of incompleteness. Most importantly, during the initial requirements gathering, clients tried to be as complete as possible in listing features, for fear of leaving out something important while they still could add scope without incurring costs. This resulted in implementation of many features that were never used in practice – an enormous waste of time and money.
• Due to lack of communication and lack of trust, negotiation was hard. Clients simply wanted as much scope as possible for as little money as possible. If legislation changed during the project execution and the scope had to change, a negotiation had to take place.

Fixing waterfall

Agile was the correct response, around the turn of the century, to the above problems. Here's how:

• Understand that the scope changing... is the norm, it is the most common thing in the world, it is not to be treated as an exception, but as a rule. ("Embrace change" is the subtitle of Kent Beck's book on XP.)
• Keep the same phases as waterfall, but do them for the smallest amount of scope possible, then repeat them. Agile is a series of tiny waterfalls.
• Because it's a series of tiny waterfalls, the whole team can work at the same time: the business analyst, the system architect, the designer, the programmer, the tester, the system administrator etc. are all working at the same time now, instead of in different phases. The pains of one can immediately be conveyed to relevant others. The system architect is immediately informed of the weaknesses in her architecture. The business analyst is immediately informed of the weaknesses in her output. The programmer immediately sees what kinds of bugs are being found by the tester. Everyone gets opportunities to improve.
• Instead of the feature creep characteristic of waterfall, agile prefers to deliver the smallest thing that has any value, the minimum viable product, as soon as possible. Even if it's just a single feature.
• Get feedback from the client and especially from users as early as possible and use that feedback in the following iterations. More likely that the right thing will be implemented. Less likely that unnecessary things will be implemented.
• Bring the client onto the development team, so there is transparency, they can actually see the hard work being done, they gain trust, and communication switches from negotiation mode to team mode.
• Even if communication fails hard at the beginning of the project, chances are it will improve and the right thing will be developed.

Maybe Waterfall can succeed for projects that are small enough or very easy to understand and describe. For instance, a device driver for a mouse. But most software projects aren't like that, and then you need something that deals with the actual challenges, and that's Agile.

About the relationship with the client, XP (Extreme Programming) seems to have been the first to alter the contract. When a client hires an XP team, the scope is not predefined anymore. The scope is variable, which the client initially doesn't like. But the contract also says that the client can cancel the project at any time without any fines – this part the client does like. Therefore, the client will keep the project going while it is worthwile, and will cancel the contract as soon as it is more expensive than the value been delivered. This is a saner way to work than the rigid waterfall contracts that existed then, with rigid scope, rigid implementation time etc.

The manifesto

Please refer to the original Agile Manifesto, which is about 5 lines long:

https://agilemanifesto.org/

It simply means that:

• processes and tools are important, but individuals and interactions are more important.
• comprehensive documentation is important, but working software is more important.
• contract negotiation is important, but customer collaboration is more important.
• following a plan is important, but responding to change is more important.

You see, it's extremely hard to disagree with anything the manifesto actually says.

For example, notice how the manifesto didn't prescribe any meetings. The meetings that developers complain about so much are usually the Scrum meetings. Therefore, they are not a part of Agile in itself. One can develop software well while organizing one's meetings in a different way.

But what some programmers really want is to develop software without communicating at all; that's impossible, and therefore, their own mistake.

Scrum

Scrum is considered an agile development methodology, appropriate for developing products of any kind, not just software. Scrum does not actually say anything specific about software development. However, it prescribes a set of meetings, which does address a tendency that software developers have, to not communicate enough (or sometimes at all) between themselves. If left unchecked, this tendency is enormously dangerous to the health of any project. We'll talk about this again below (XP).

The Scrum Guide (2009-2020) is another document that is very brief (12 pages) and easy to read. Again, you might be surprised at the things it doesn't say. For instance, Kanban and Planning Poker are not necessarily a part of Scrum. Scrum does not prescribe how the product backlog should be organized or how the team actually decides what to include in the next sprint. You might be surprised at how democratic it really is, if you read it.

I have worked in several bad Scrum implementations, each with their own troubles. However, I do know from experience that Scrum can work wonderfully well, because I have been part of a good Scrum implementation. Its democratic nature kept the entire team quite enthusiastic about their work.

Scrum is a humanistic revolution in the workplace. It inverts the hierarchy. If in the 80s you had quite bossy bosses bossing workers around, Scrum says, bosses are out. Scrum teams have no boss; there is only a facilitator, who is supposed to give workers freedom to do the work the way they want to do it. The role of the facilitator is actually to help the workers, for instance by removing obstacles (institutional, physical etc.) that hinder their best work. This creates an engaged team where before you only had begrudging employees. The Scrum team is self-managing.

"But in my experience Scrum is a tool of domination", you say. Well, did you experience actual Scrum, or a bastardized "version" that actually contradicts its principles?

If you are going to criticize Agile or Scrum, first understand what each actually says. Most often, you'll realize your trouble is with the implementations you've had the misfortune of experiencing.

If you pay attention to the history of Scrum, you realize there are at least 2 Scrums:

• The 1986 one, from the paper "The New New Product Development Game" by Hirotaka Takeuchi and Ikujiro Nonaka;
• The famous one, from Schwaber and Sutherland in 1995. Both authors are also signataries of the 2001 Manifesto for Agile Software Development. But the Scrum Guide only appeared in 2009.

It is important to notice that the lineage of thought is different for Scrum – it originated in Toyota, not in software development. Quoting the above Wikipedia article about the Japanese original Scrum:

Based on case studies from manufacturing firms in the automotive, photocopier, and printer industries, the authors outlined a new approach to product development for increased speed and flexibility. They called this the rugby approach, as the process involves a single cross-functional team operating across multiple overlapping phases, in which the team "tries to go the distance as a unit, passing the ball back and forth". The authors later developed scrum in their book, The Knowledge Creating Company.

What I can see Scrum has in common with Agile is the humanistic aspect:

• individuals and interactions over processes and tools
• collaboration over negotiation – this time within the team.

Extreme Programming (XP)

XP is the actual first agile methodology for software development specifically. This is the one that was made by programmers, for programmers. It was developed by Kent Beck and his team from 1996 to 1999, when he wrote Extreme Programming Explained. That team included these guys who also signed the Agile Manifesto:

• Kent Beck, inventor of XP
• Ward Cunningham, the inventor of the wiki
• Ron Jeffries

Most of the ideas in the Agile Manifesto are already found in XP.

XP described most of the software development practices that won and are in use today:

• writing automated tests and using them to get the courage to refactor existing code freely, without worrying about breaking everything
• refactoring before implementing a new feature, so it becomes easier to implement
• delaying decisions when possible
• pair programming to improve team communication, disseminate knowledge amongst team members, and to severely improve code quality, killing bugs immediately
• writing only documentation that you actually need and can keep up-to-date

XP was revolutionary, especially when compared to Almighty Thud methodologies which emphasized writing lots of documentation, which were the norm in the Waterfall era. (The Almighty Thud is the name for the noise made by the volume of paper documentation hitting one's desk.)

In fact, XP is still revolutionary, inasmuch as you haven't yet personally practiced its propositions.

Software is a tamagotchi

The main conflict I see in software development is the following: Agile promises continuous delivery of value (something like delivering features all the time), but software development simply isn't like that.

Expecting a development team to deliver only features all the time is as realistic as expecting a human being to be of service to their significant other all the time. The reality is, the human and the team need some time to themselves.

Business people start with this idea that software is a project with a beginning and an end. "I will develop an app, then sit back and profit." Nothing could be further…

Reality is like Chacon said:

Software is a tamagotchi. It has its own needs, that must be tended to.

The problem is, business people cannot see the virtual pet or its demands, if they cannot program. A programmer can try to explain it to them, but they get bored quickly.

This tamagotchi is a mythical creature. Only little children can see E. T., grownups cannot. And only developers can see the needs of the software; they are invisible to the marketing team.

This past week I upgraded packages in our Linux servers because of the recently discovered openssh vulnerability. If you have servers, you have to be on top of security, there is no alternative.

— Yeah, but as a business person, I absolutely CAN see the value and importance in that work, since not having our users' passwords stolen is a major component of their satisfaction.

Okay, then I suppose sometimes you can see the tamagotchi demanding we do things that our users are completely unaware of.

Let me try another example then. We really have to do some work on the way we use our asynchronous queue system. You see, things like sending emails, talking to external APIs, anything that isn't instantaneous... these things are done in separate processes, in a queue. That queue is known as FIFO, which means First In, First Out, which means first come, first served. The queue is a buffer for tasks that the app needs to do. But the queue has certain features that we need to start using in order not to have problems in it, which would be a terrible situation.

— That's too technical for me, you seem to be saying words. If I were to prioritize that work, I think I would actually never prioritize it, I think, because I have features that our users actually want, and those I do understand.

…and this is an example of the virtual pet dying of starvation, because the parents didn't understand its importance.

Technical debt

It is a terrible thing, when instead of the whole team choosing what needs to be done in the next sprint, it gets dictated by a non-technical Product Owner who leaves technical debt on the wall forever gathering spiders' webs, until it falls on the ground and gets swept away by someone who doesn't think they are important.

A good rule for solving technical debt had already been proposed in XP: refactor before you implement a new feature. This means, remove the technical debt that affects the feature you are about to implement. It also means, leave alone the technical debt that does not affect the code you need to change. Some technical debt exists in code that never gets changed, and that is okay if the code is working fine.

By the way, bugs are not technical debt, bugs are bugs. Bugs annoy users, technical debt "only" annoys developers. Generally, solving bugs is the most important priority, because if you leave bugs alone, they start compounding. Then when you finally decide to kill them, you can't tell anymore where one starts and ends, because they have become this multi-headed monster. Nobody can understand the behavior of a buggy system; a sane mind can only understand sane behavior.

However, annoying developers is a terrible thing too, and that is what business people don't understand.

Clean code

There's a second document about Agile, also very brief, which tries to clarify the Agile Manifesto. It's the 12 Principles behind the Agile Manifesto. Again I invite you to read them.

Here's a bit I would like to emphasize:

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Continuous attention to technical excellence and good design enhances agility.

This means, agile is not supposed to be a tool against developers. This means the maintainability of the code is valuable, and important to agility.

Any psychologist will tell you, before you can love other people, you need to actually love yourself.

As a developer, each day you must do something to improve your own life. Nobody else is going to do that for you. They will even have difficulty understanding that an annoyed programmer is a much slower programmer, naturally and for good reason. It's hard to put something in production when you can't get past development.

Martin Fowler, a famous and respected British author who writes the best descriptions of ideas in software development, while offering the sanest opinions, went through the trouble of cataloguing and describing the various refactorings, in his famous 1999 book Refactoring. He is also one of the first signataries of the Agile Manifesto.

Why would he write Refactoring if technical debt were not important enough to fix???

Oh, and here is his description of technical debt.

Robert C. Martin, a very important author whose name you also find under the Agile Manifesto, is sort of saying the same thing, when he says, the only way to go fast, is to go well.

A large application cannot be built without lots of discipline. A mature programmer knows this and takes pride in the discipline, just like a practitioner of martial arts or a musician.

Robert C. Martin, in fact, goes beyond the XP rule. He notices that people are about to start dying due to badly written software, or maybe already are dying. So he says, when the thing works, do not stop and deliver it. When the thing works, you are only half done. Now you have to clean it up. Remove all the technical debt, make it easy for other programmers to understand. Make it easy even for you yourself to understand it after 2 months! Remove the useless tests, finish writing the good ones. Refactor the spaghetti, rename the badly named functions. Treat your program like a poem.

If the company places excessive pressure on developers, it is an engineer's duty to be firm and say no, especially if people's health is directly or indirectly at stake. This is the ultimate test of a true professional. You do know that "I was just following orders" does not convince. Show some character strength when your turn comes.

If developers do not self-police in this way, people will die due to buggy programs, and then the legislators will legislate software development, and the result will be much worse. This is what Martin says, and it's hard to disagree.

As a developer, do love your users, but love yourself first. You can't be of use to your users, or the company, or your family, if your software becomes unmaintainable.

In other words, no matter how much you love your user, you can't be used by him all day long!

When a musician gets popular, she often needs to put her foot down and say "no, only 2 shows a week maximum and that's final". Otherwise there's no time to hone her craft. These companies always want you for what you already know, but never want you to spend any time learning what you need to learn to stay relevant. It's up to you to draw the line.

According to Woody Zuill, teams and managers should spend the time to make the work easy to do.

Allen Holub has a heuristics page in which he briefly explains agile in his own words. At the time I am writing, number 15 reads: "Quality is not negotiable. (This rule applies to all aspects of quality, not just testing.)"

Conclusion

When bad managers use Scrum against developers and basic development needs, that is in fact against Agile principles.

A solution for our woes will be found in a proper implementation of Agile ideas – not in mistakenly escaping those ideas as if something better existed out there.

Mind the gurus you choose. Digital influencers can be entertaining and arouse emotions, but some of their advice is terribly misguided. To them, all the names in this article are preferable.