4. Learning and Adapting
Introduction
Software development is a continuous learning activity between the customer who knows the business domain and the developers who know the software. Since the software product needs to combine the two areas of knowledge, the customers must teach the software developers about the business domain and the software developers must teach the customer enough about software so that they can better express their requirements. This learning activity is done throughout the project, having both the business domain experts and the software developers communicating and collaborating all the time, teaching each other what is needed in order to build the best software.
Circle of life – learning and adapting
In most occasions, at the beginning of a software project, the customer doesn’t really know what he wants or cannot express very well what he wants. The intangible, virtual nature of software is often the main factor in this problem. Traditional, predictability based processes, which need to write all the customer requirements at first, are exposed to high risks in projects where the customer cannot express his needs from the beginning. Not knowing what they want at first, means that the customer will want changes later and the cost of change is very high late in development, in waterfall based processes:
[Cost of change in traditional processes, by Scott W. Ambler]
Agile methodologies have acknowledged the problem of having customers that cannot express their whishes at first, and have built, at the base of their processes, a way of development that allows continuous learning and adapting throughout the project lifecycle, handling very well the cost of change:
[Cost of change in XP, by Scott W. Ambler]
In “Extreme Programming: Embrace Change” [], Kent Beck, explains how XP handles the problem of having customers that don’t know what they want and want changes late in development:
For decades, programmers have been whining, "The customers can't tell us what they want. When we give them what they say they want, they don't like it." This is an absolute truth of software development. The requirements are never clear at first. Customers can never tell you exactly what they want.
The development of a piece of software changes its own requirements. As soon as the customers see the first release, they learn what they want in the second release...or what they really wanted in the first. And it's valuable learning, because it couldn't have possibly taken place based on speculation. It is learning that can only come from experience. But customers can't get there alone. They need people who can program, not as guides, but as companions.
What if we see the "softness" of requirements as an opportunity, not a problem?
In his book “Agile and iterative development: A manager’s guide”, Craig Larman shows a very good sample of how, iteration by iteration, requirements evolve:
[Craig Larman, 2004]
The iterative and incremental nature of the agile methodologies allows the customers and the developers to continually adapt the software to their needs. Kent Beck compares software development with driving a car:
We need to control the development of software by making many small adjustments, not by making a few large adjustments, kind of like driving a car. This means that we will need the feedback to know when we are a little off, we will need many opportunities to make corrections, and we will have to be able to make those corrections at a reasonable cost.
The software process is a continuous cycle following a few simple steps: the developers and the customer decide what to do in the next iteration, the developers do it and then they show the result to the customer, who can now see the software and learn from it. The customers can now decide what needs to be corrected or improved and they can now give feedback about the product, that makes the programmers understand better what the customers really want. After a cycle like this, everything is taken from the beginning again. This is a continuous learning activity between the customer and the developers. Ron Jeffries calls this “the circle of life”, saying that:
On an XP project, the customer defines business value by writing stories, and the programmers implement those stories, building business value. But there’s an important caveat: on an XP project, the programmers do what the customer asks them to do!
Every time we go around the circle, we learn. Customers learn how valuable their proposed features really are, while programmers learn how difficult the features really are. We all learn how long it takes to build the features we need.
As important it is to develop closely with the customer and learn and adapt after every iteration, in order to improve the software, it is more important that the developers collect feedback from its real users, every release when the software is put in production and is used daily. Throughout the development of a release, the learning and adapting is done with the customer together, but nothing can give better feedback than real use of the software after a software release or delivery has been made to the real end users.
Development and production
Mary and Tom Poppendieck consider that one of the most important principles in lean thinking (adapted to software development) is amplifying learning, and that this is done as described above, using the main tools: iterations and feedback.
They also manage to show that it is very important to make the distinction between software development and product manufacturing, saying:
Think of development as creating a recipe and production as following the recipe. … Developing a recipe is a learning process involving trial and error. You would not expect an expert chef's first attempt at a new dish to be the last attempt. In fact, the whole idea of developing a recipe is to try many variations on a theme and discover the best dish
Showing the main differences between development and production
Development - Designs the Recipe
- Quality is fitness for use
- Variable results are good
- Iteration generates value
Production - Produces the Dish
- Quality is conformance to requirements
- Variable results are bad
- Iteration generates waste (called rework)
[Mary and Tom Poppendieck, 2003]
Not seeing these very important differences means not seeing development as a continuous learning and adapting process, thus working against nature, increasing the risks of failure in software projects.
A game of invention and communication
Alistair Cockburn defines software development as in Agile Software Development []:
Software development is a (resource limited) cooperative game of invention and communication. The primary goal is to deliver useful, working software. The secondary goal, the residue of the game, is to set up for the next game. The next game may be to alter and replace the system or to create a neighboring system
He expresses the need for cooperation between developer and customers as in a group game, and the need between the participants at the game to learn the problem and invent or imagine a solution for it, constantly readapting it to fit the needs.
3.3 Reflective improvement
One of the most important aspects of the agile movement is that its followers constantly need to look back at their activity, and learn from what they do about what is beneficial and what isn’t, and, based on this information, adapt and improve their working process. The need to improve the process is very well described by the last principle of the agile manifesto:
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Alistair Cockburn, describes a very pragmatic way to put the self improving technique in practice, in the book “Crystal Clear” []. At the end of each iteration, the team gets together and lists the activities they did in the last iteration, dividing them into two columns: to keep (activities that worked well and should be kept in the process) and to drop (activities, practices that didn’t work so well, and should not be put in practice anymore). Besides the two columns, a third column is added: “to try”, where ideas about different practices that could be tried are added by the team. Alistair calls this technique: reflective improvement.
The intervals at which the team reflects should not be very long, and Alistair suggests twice per iteration, once in the middle of the iteration when things can be improved as the iteration progresses, and one at the end of the iteration, to reflect on the entire iteration, including the delivery to the client and his satisfaction, called “post mortem” reflection workshop.
The activities and techniques discussed in a reflection workshop inside the team do not necessarily need to be from the past iteration or related to a project. They can be general conventions used throughout a larger base than the actual team like code conventions or database versioning and the people participating in them do not necessarily need to be from the team, so this kind of sessions could be done even with the customer to improve the whole collaboration and development process together.
Eliminating waste and cutting overhead
In order for a team to be able to drop some techniques that they are using, they need first to be aware that those techniques are not useful. Although this sounds incredibly simple, it proves to be one of the hardest to put in practice techniques. Seeing what generates waste is a very difficult activity and needs to be discussed in detail starting from what waste is.
Mary and Tom Poppendieck, have adapted a method to see waste in software development, from lean production manufacturing, that is based on Taichi Ohno’s first steps at
| The 7 wastes in lean manufacturing | The 7 wastes of software development |
| Inventory Extra Processing Overproduction Transportation Waiting Motion Defects | Partially Done Work Extra Processes Extra Features Task Switching Waiting Motion Defects |
In many occasions we develop code, especially when designing software, to allow flexibility for future changes. Some of the investment in the flexible design is well paid off, but in other occasions, those changes never occur, and the investment in the flexibility of the design is waste.
Following religiously a software process does not guarantee success. However, when software development fails, in many cases, the developers and the management come to the conclusion that the process wasn’t followed and they decide to make it stricter: more documentation is produced, more meetings are scheduled, the plans and designs go in further details. This can be good in some cases, but extra process overhead is one of the biggest problems in software development today. Programmers that work for 6 hours a day on writing documents that comply with the process and two hours on actually producing software is one of the best hidden wastes in software development, especially in large organizations.
Moving programmers from one project to another, requires time for them to get used to the new project. If that project is big, then understanding takes a lot of time, and if the task performed by the programmer moved to the project is small it does not cover the initial investment in understanding and, even worse, because the software wasn’t understood properly the programmers decrease the internal quality of the project, making it harder to extend and maintain.
One of the biggest problems with waste is that most programs have features that are never used. These contribute essentially to the cost, length and complexity of a software project, and not using them is just waste.
Self improving sessions
Back to work:
Some time ago, we started to use a new open source ORM on .NET, called NH. It was a fairly small (2.5 months) and non complex project, that was built by me and another colleague, let’s call him John. I was needed for the first 3 iterations (6 weeks) to help get the project started then it would be continued by John. The project was delivered on time for the customer and it still works. Then John and another colleague, Michael started a new web project, quite similar to the previous and developed it for two months. Then Michael and another colleague, George, started the third project.
On one discussion outside the company, Michel and George were complaining about the ORM saying that in many situations it proved to be an overhead. Suddenly, I detected a potential problem. I had been using it for the same period and my conclusion was exactly the opposite. Listening further, we decided to take a look in the code, and it was then that I realized that some parts of it were completely misused, becoming an overhead. Going back to the second project, I noticed the same mistake there. And, quite surprisingly in the first project also. I realized that one mistake, made by John, in using NH, was considered the right way to go by Michael and later by George resulting in much more code then it was actually supposed to be and the impression that it wasn’t a good technology.
I decided that I needed to do something about this, so I said to the entire team that the next day at 2pm everyone needs to be available for a meeting to sort out the problem; however I warned everyone that it wasn’t going to be a conventional meeting. When everyone came, I gave them a document, of about 40 pages that described in detail how NH really worked and I asked everyone to read it. While they were reading I made a list of questions to see if they understood the technology and we were supposed to discuss the responses in the end. When one finished reading I sent the questions by email expecting the answers in the same way. Of course we were all collocated and everyone had his computer. When all the answers came I was surprised that they were 99% right.
So, after 2.5 hours, that might be considered lost, because instead of working we read about a technology that was not used by everyone at the time, everyone started to understand it. If these 2.5 hours would have been done at the beginning of the first project, this might have saved days or maybe weeks of wasted resources because the technology wasn’t properly understood by someone and these misunderstandings were taken by the others as proper ways to use the technology.
We decided that this kind of “ school like sessions” could have tremendous potential also in our attempt to become more agile, so we started with Extreme Programming, reading, answering questions and discussing a few pages every day, having information and knowledge spread much faster than ever before, which later avoided many overheads and wasted resources.
No comments:
Post a Comment