CHAPTER 4: Learning and adapting

4. Learning and Adapting

Introduction

Software development is a continuous learning activity between the customer who knows the business domain and the developers who know the software. Since the software product needs to combine the two areas of knowledge, the customers must teach the software developers about the business domain and the software developers must teach the customer enough about software so that they can better express their requirements. This learning activity is done throughout the project, having both the business domain experts and the software developers communicating and collaborating all the time, teaching each other what is needed in order to build the best software.

Circle of life – learning and adapting

In most occasions, at the beginning of a software project, the customer doesn’t really know what he wants or cannot express very well what he wants. The intangible, virtual nature of software is often the main factor in this problem. Traditional, predictability based processes, which need to write all the customer requirements at first, are exposed to high risks in projects where the customer cannot express his needs from the beginning. Not knowing what they want at first, means that the customer will want changes later and the cost of change is very high late in development, in waterfall based processes:

[Cost of change in traditional processes, by Scott W. Ambler]

Agile methodologies have acknowledged the problem of having customers that cannot express their whishes at first, and have built, at the base of their processes, a way of development that allows continuous learning and adapting throughout the project lifecycle, handling very well the cost of change:

[Cost of change in XP, by Scott W. Ambler]

In “Extreme Programming: Embrace Change” [], Kent Beck, explains how XP handles the problem of having customers that don’t know what they want and want changes late in development:

For decades, programmers have been whining, "The customers can't tell us what they want. When we give them what they say they want, they don't like it." This is an absolute truth of software development. The requirements are never clear at first. Customers can never tell you exactly what they want.

The development of a piece of software changes its own requirements. As soon as the customers see the first release, they learn what they want in the second release...or what they really wanted in the first. And it's valuable learning, because it couldn't have possibly taken place based on speculation. It is learning that can only come from experience. But customers can't get there alone. They need people who can program, not as guides, but as companions.

What if we see the "softness" of requirements as an opportunity, not a problem?

In his book “Agile and iterative development: A manager’s guide”, Craig Larman shows a very good sample of how, iteration by iteration, requirements evolve:

[Craig Larman, 2004]

The iterative and incremental nature of the agile methodologies allows the customers and the developers to continually adapt the software to their needs. Kent Beck compares software development with driving a car:

We need to control the development of software by making many small adjustments, not by making a few large adjustments, kind of like driving a car. This means that we will need the feedback to know when we are a little off, we will need many opportunities to make corrections, and we will have to be able to make those corrections at a reasonable cost.

The software process is a continuous cycle following a few simple steps: the developers and the customer decide what to do in the next iteration, the developers do it and then they show the result to the customer, who can now see the software and learn from it. The customers can now decide what needs to be corrected or improved and they can now give feedback about the product, that makes the programmers understand better what the customers really want. After a cycle like this, everything is taken from the beginning again. This is a continuous learning activity between the customer and the developers. Ron Jeffries calls this “the circle of life”, saying that:

On an XP project, the customer defines business value by writing stories, and the programmers implement those stories, building business value. But there’s an important caveat: on an XP project, the programmers do what the customer asks them to do!

Every time we go around the circle, we learn. Customers learn how valuable their proposed features really are, while programmers learn how difficult the features really are. We all learn how long it takes to build the features we need.

As important it is to develop closely with the customer and learn and adapt after every iteration, in order to improve the software, it is more important that the developers collect feedback from its real users, every release when the software is put in production and is used daily. Throughout the development of a release, the learning and adapting is done with the customer together, but nothing can give better feedback than real use of the software after a software release or delivery has been made to the real end users.

Development and production

Mary and Tom Poppendieck consider that one of the most important principles in lean thinking (adapted to software development) is amplifying learning, and that this is done as described above, using the main tools: iterations and feedback.

They also manage to show that it is very important to make the distinction between software development and product manufacturing, saying:

Think of development as creating a recipe and production as following the recipe. … Developing a recipe is a learning process involving trial and error. You would not expect an expert chef's first attempt at a new dish to be the last attempt. In fact, the whole idea of developing a recipe is to try many variations on a theme and discover the best dish

Showing the main differences between development and production

Development - Designs the Recipe

• Quality is fitness for use
• Variable results are good
• Iteration generates value

Production - Produces the Dish

• Quality is conformance to requirements
• Variable results are bad
• Iteration generates waste (called rework)

[Mary and Tom Poppendieck, 2003]

Not seeing these very important differences means not seeing development as a continuous learning and adapting process, thus working against nature, increasing the risks of failure in software projects.

A game of invention and communication

Alistair Cockburn defines software development as in Agile Software Development []:

Software development is a (resource limited) cooperative game of invention and communication. The primary goal is to deliver useful, working software. The secondary goal, the residue of the game, is to set up for the next game. The next game may be to alter and replace the system or to create a neighboring system

He expresses the need for cooperation between developer and customers as in a group game, and the need between the participants at the game to learn the problem and invent or imagine a solution for it, constantly readapting it to fit the needs.

3.3 Reflective improvement

One of the most important aspects of the agile movement is that its followers constantly need to look back at their activity, and learn from what they do about what is beneficial and what isn’t, and, based on this information, adapt and improve their working process. The need to improve the process is very well described by the last principle of the agile manifesto:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Alistair Cockburn, describes a very pragmatic way to put the self improving technique in practice, in the book “Crystal Clear” []. At the end of each iteration, the team gets together and lists the activities they did in the last iteration, dividing them into two columns: to keep (activities that worked well and should be kept in the process) and to drop (activities, practices that didn’t work so well, and should not be put in practice anymore). Besides the two columns, a third column is added: “to try”, where ideas about different practices that could be tried are added by the team. Alistair calls this technique: reflective improvement.

The intervals at which the team reflects should not be very long, and Alistair suggests twice per iteration, once in the middle of the iteration when things can be improved as the iteration progresses, and one at the end of the iteration, to reflect on the entire iteration, including the delivery to the client and his satisfaction, called “post mortem” reflection workshop.

The activities and techniques discussed in a reflection workshop inside the team do not necessarily need to be from the past iteration or related to a project. They can be general conventions used throughout a larger base than the actual team like code conventions or database versioning and the people participating in them do not necessarily need to be from the team, so this kind of sessions could be done even with the customer to improve the whole collaboration and development process together.

Eliminating waste and cutting overhead

In order for a team to be able to drop some techniques that they are using, they need first to be aware that those techniques are not useful. Although this sounds incredibly simple, it proves to be one of the hardest to put in practice techniques. Seeing what generates waste is a very difficult activity and needs to be discussed in detail starting from what waste is.

Mary and Tom Poppendieck, have adapted a method to see waste in software development, from lean production manufacturing, that is based on Taichi Ohno’s first steps at Toyota in the 40s:

The 7 wastes in lean manufacturing	The 7 wastes of software development
Inventory Extra Processing Overproduction Transportation Waiting Motion Defects	Partially Done Work Extra Processes Extra Features Task Switching Waiting Motion Defects

In many occasions we develop code, especially when designing software, to allow flexibility for future changes. Some of the investment in the flexible design is well paid off, but in other occasions, those changes never occur, and the investment in the flexibility of the design is waste.

Following religiously a software process does not guarantee success. However, when software development fails, in many cases, the developers and the management come to the conclusion that the process wasn’t followed and they decide to make it stricter: more documentation is produced, more meetings are scheduled, the plans and designs go in further details. This can be good in some cases, but extra process overhead is one of the biggest problems in software development today. Programmers that work for 6 hours a day on writing documents that comply with the process and two hours on actually producing software is one of the best hidden wastes in software development, especially in large organizations.

Moving programmers from one project to another, requires time for them to get used to the new project. If that project is big, then understanding takes a lot of time, and if the task performed by the programmer moved to the project is small it does not cover the initial investment in understanding and, even worse, because the software wasn’t understood properly the programmers decrease the internal quality of the project, making it harder to extend and maintain.

One of the biggest problems with waste is that most programs have features that are never used. These contribute essentially to the cost, length and complexity of a software project, and not using them is just waste.

Self improving sessions

Back to work:

Some time ago, we started to use a new open source ORM on .NET, called NH. It was a fairly small (2.5 months) and non complex project, that was built by me and another colleague, let’s call him John. I was needed for the first 3 iterations (6 weeks) to help get the project started then it would be continued by John. The project was delivered on time for the customer and it still works. Then John and another colleague, Michael started a new web project, quite similar to the previous and developed it for two months. Then Michael and another colleague, George, started the third project.

On one discussion outside the company, Michel and George were complaining about the ORM saying that in many situations it proved to be an overhead. Suddenly, I detected a potential problem. I had been using it for the same period and my conclusion was exactly the opposite. Listening further, we decided to take a look in the code, and it was then that I realized that some parts of it were completely misused, becoming an overhead. Going back to the second project, I noticed the same mistake there. And, quite surprisingly in the first project also. I realized that one mistake, made by John, in using NH, was considered the right way to go by Michael and later by George resulting in much more code then it was actually supposed to be and the impression that it wasn’t a good technology.

I decided that I needed to do something about this, so I said to the entire team that the next day at 2pm everyone needs to be available for a meeting to sort out the problem; however I warned everyone that it wasn’t going to be a conventional meeting. When everyone came, I gave them a document, of about 40 pages that described in detail how NH really worked and I asked everyone to read it. While they were reading I made a list of questions to see if they understood the technology and we were supposed to discuss the responses in the end. When one finished reading I sent the questions by email expecting the answers in the same way. Of course we were all collocated and everyone had his computer. When all the answers came I was surprised that they were 99% right.

So, after 2.5 hours, that might be considered lost, because instead of working we read about a technology that was not used by everyone at the time, everyone started to understand it. If these 2.5 hours would have been done at the beginning of the first project, this might have saved days or maybe weeks of wasted resources because the technology wasn’t properly understood by someone and these misunderstandings were taken by the others as proper ways to use the technology.

We decided that this kind of “ school like sessions” could have tremendous potential also in our attempt to become more agile, so we started with Extreme Programming, reading, answering questions and discussing a few pages every day, having information and knowledge spread much faster than ever before, which later avoided many overheads and wasted resources.

CHAPTER 3 (continued) : 3.4 Collaborating

3.4 Collaborating

Introduction

One of the values of the agile manifesto states: “Customer collaboration over following a contract” being completed then in the principles by: “Business people and developers must work together daily throughout the project”. It is very clear that day by day collaboration, between team members or between programmers and customers can deliver better software and, as a result, close collaboration stands at the base of all agile methodologies.

Why is collaboration important?

Building software is a continuous learning process between the customers that know the business constraints and the programmers who can develop the software. The customers must learn about software, about how software can be built, about what can be done and what can’t, about the cost of the different pieces of functionality because he is the one to “steer” the software direction to what he needs for his business. To build the right software, the programmers must learn about the business domain of the customer. This learning can only be done by working together daily.

Robert C. Martin said in “Agile Software Development, Principles, Practices and Patterns” [] that there is this tendency for managers to tell the programmers what needs to be done then go away for as long as they think it is necessary for the software to be built, then come back and find everything working. This tendency is in human nature, and it exists also in the case of the customers; however it was proven over and over that having everything written down in a functional specification document, signed off at the beginning can lead to misunderstanding the software’s purpose, from the client point of view.

Fear

Delivering software that fails to comply with the business needs of the customer contributes to building an atmosphere of distrust between software developers and the customers. This lack of trust is then tried to be compensated by very detailed contracts on what can and what cannot be done in future projects.

Customers are afraid that

• They won't get what they asked for.

• They'll ask for the wrong thing.

• They'll pay too much for too little.

• They must surrender control of their career to techies who don't care.

• They won't ever see a meaningful plan.

• The plans they do see will be fairy tales.

• They won't know what's going on.

• They'll be held to their first decisions and won't be able to react to changes in the business.

• No one will tell them the truth.

Developers are afraid, too. They fear that

• They will be told to do more than they know how to do.

• They will be told to do things that don't make sense.

• They are too stupid.

• They are falling behind technically.

• They will be given responsibility without authority.

• They won't be given clear definitions of what needs to be done.

• They'll have to sacrifice quality for deadlines.

• They'll have to solve hard problems without help.

• They won't have enough time to succeed.

Unacknowledged Fear Is the Source of All Software Project Failures

If these fears are not put on the table and dealt with, then developers and customer each try to protect themselves by building walls.

[Fowler , Beck – Planning Extreme Programming, 2001]

Contracts

Protecting by building walls, usually means contracts. Trying to replace trust and close collaboration with contracts is not solving the problem, but amplifying it. Mary and Tom Poppendieck describe why trust cannot be replaced very well with samples from car manufacturing and from software, enumerating the different types of contracts and showing the flaws they have when it comes to “enforcing” trust.

Can the parties involved in a software project as buyer and as vendor collaborate efficiently without each following their own interests? Because of this great fear, contracts are signed and expected later to replace trust between the companies involved on each side. If as quoted in Lean Software Development, trust means: “one party’s confidence that the other party … will fulfill its promises and will not exploit its vulnerabilities”, then contracts are considered by very many people, a way to enforce this.

Let’s study a little the contract types, used mostly between software companies and examine how trust can be enforced by them or how they can obtain exactly the opposite because of the contracts:

1. Fixed price contracts: The most commonly used type of contract, seems to give the customer the faith that the cost won’t be bigger then the one planned, but this means that the burden is moved on the shoulders of the software vendor, who will try to find different ways to compensate its eventual losses (if any) by overcharging for changes. They tend to push an atmosphere of distrust between the parties, each trying to follow their interest and thus favoring the failure of the software projects
2. Time and materials contracts: this type of contract deals better with unpredictability and uncertainty but simply shifts the risk from the vendor to the buyer. The longer the contract takes to complete the more the customer will pay, encouraging in a way the vendor to charge more for the work. The customer is always on the pressure that the things might be done faster and this favors ,just as the fixed price contract, an atmosphere of distrust
3. Multistage contracts: these types of contracts try to merge together the fixed price and time and materials contract types benefits, minimizing the risks. Usually a big fixed priced contract is divided into a multiple succeeding stage contracts governed by another contract. There is a certain risk in this approach also: if after each small stage contract is delivered, the contract is fully renegotiated, then it can become very expensive and it will not bring the two parties close together.
4. Target cost contracts: this type is the hardest to put in practice, but the most fair for both parties as well, which means that it favors trust. They are structured in such a way that the total cost, including changes – is a joined responsibility of the customer and vendor. This enforces them to collaborate and take decisions together, negotiating so that the target cost is met.

Although, some types of contracts can be seriously flawed when it comes to software development and to building programs that have value for the customers, it has been proven that they can be adapted to work well in software as long as they do not try to replace trust. Agile methodologies tend to favor the types of contracts that are aimed at building trust rather than building distrust. Robert C. Martin gives a very good example of an agile contract:

As an example of a successful contract, in 1994 I negotiated a contract for a large, multi-year, half-million-line project. We, the development team, were paid a relatively low monthly rate. Large payouts were made to us when we delivered certain large blocks of functionality. Those blocks were not specified in detail by the contract. Rather the contract stated that the payout would be made for a block when the block passed the customer’s acceptance test. The details of those acceptance tests were not specified in the contract.

During the course of this project we worked very closely with the customer. We released the software to him almost every Friday. By Monday of Tuesday of the following week he would have a list of changes for us to put into the software. We would prioritize those changes together, and then schedule them into subsequent weeks. The customer worked so closely with us that acceptance tests were never an issue. He knew when a block of functionality satisfied his needs because he watched it evolve from week to week.

The requirements for this project were in a constant state of flux. Major changes were not uncommon. There were whole blocks of functionality that were removed, and others that were inserted. And yet the contract, and the project, survived and succeeded. The key

to this success was the intense collaboration with the customer; and a contract that governed that collaboration rather than trying to specify the details of scope and schedule for a fixed cost.

[Robert C. Martin 2005]

The agile methodologies come with a series of practices that are aimed at building trust between the vendor and the buyer of the software, frequent delivery being the most important of them, because the customer can see the software, at small intervals, with the planned features, built and working. Risks are also addressed by having the most important features (from the client’s point of view) developed first and by allowing him to make mistakes, to change his mind, to figure out how things are really done later after he sees what the software would look and feel.

Customer and programmer rights

Ron Jeffries, in “Extreme Programming Installed” [] and Martin Fowler and Kent Beck in “Planning Extreme Programming” [], built a list of rights for all the three parties usually involved in a software project, which, if known and respected from the beginning, can enhance collaboration.

Customer Bill of Rights

• You have the right to an overall plan, to know what can be accomplished when and at what cost.

• You have the right to get the most possible value out of every programming week.

• You have the right to see progress in a running system, proven to work by passing repeatable tests that you specify.

• You have the right to change your mind, to substitute functionality, and to change priorities without paying exorbitant costs.

• You have the right to be informed of schedule changes, in time to choose how to reduce the scope to restore the original date. You can cancel at any time and be left with a useful working system reflecting investment to date.

Programmer Bill of Rights

• You have the right to know what is needed, with clear declarations of priority.

• You have the right to produce quality work at all times.

• You have the right to ask for and receive help from peers, managers, and customers.

• You have the right to make and update your own estimates.

• You have the right to accept your responsibilities instead of having them assigned to you.

Customer collaboration

In “Extreme Programming Explained: Embrace Change” [], Kent Beck says:

XP planning assumes that the customer is very much part of the team, even if the customer works for a different company and the rest of the team works for a contractor. The customer must be part of the team because his role is far too important to leave to an outsider. Any (XP) project will fail if the customer isn't able to steer.

So the customer's job is a huge responsibility. All the best talent and technology and process in the world will fail when the customer isn't up to scratch. Sadly, it's also a role on which we can't offer particularly sage advice. After all, we're nerds, not business people. But here's what we know for sure.

[Kent Beck, 1999]

This very much shows the role of the customer in an agile project, and how close collaboration is actually being done, with each party having their responsibilities and rights, and collaborating to “steer” and implement the project successfully.

The most obvious way in which the customers and the programmers collaborate, is agile planning, which is done together. SCRUM iteration or sprint planning summarizes in a way the whole concept behind agile planning that is also used in all the other agile “flavors”. The customers and the programmers sit together, starting with the customer explaining in the first part of their meeting where he wants to go, showing the “shopping” list he wants for the next sprint and explaining what each item means, so that the programmers can come up with the cost (estimated time) for those items in the second part of the meeting. In the case that the cost is over the budget (30 days in a sprint) then based on the priorities of the customer some of the low priority items are dropped, so that the cost matches the budget. In case the cost is under, then the customer is asked to come up with other items, that are the next most important to him, so the gap between cost and budget is filled.

One unwanted possibility (but, since agile methodologies are very pragmatic, it is taken into account) is when the team realizes that they won’t be finishing in time, with the next iteration. Since honesty sits at the base of agile principles, then the customer is informed right away and shown the situation. He is then asked to pick one or more of the planned features, and “drop” it, so that the remaining can be delivered successfully. In this way, the crisis is solved fast and together, whereas the alternative: lying would be much more costly. It is important to remember that in this “negotiating scope” session, for example , 8, 100% finished features will be delivered out of 10 planned, rather then 10, 80% finished features that would be unusable. The “dropped” features will be then planned for the next iteration.

The two methods described above are incredibly efficient at reducing the risk of a project failure since the top most important features are built and delivered first, thus showing how close collaboration between the customer and the programmers can lead to success.

CHAPTER 3: Communicating and Collaborating

3.1 Introduction

Communication is considered to be the main factor in a project’s success or failure. Alistair Cockburn calls software development “a game of communication and invention”, in his book “Agile Software Development”, where a big part is dedicated to explaining communication problems and how communication influences the development of a software project. In all agile methodologies, communication and close collaboration with the customer is emphasized as a “must do” practice; in XP two of the five values are communication and feedback, in SCRUM’s values we can find communication, Crystal comes with osmotic communication between team members and with early access to expert users and so on.

Bad communication between the different people involved directly in a software project, can create a lot of overhead and waste important resources, and in many cases can cause the failure of the project. For this reason, all software methodologies have developed more standards for communicating, formal and informal, while working on a project, between the different stakeholders.

Communication can be divided in: communication inside the development team and communication outside the development team, with the customer and with the upper management.

3.2 Communicating with the customer

3.2.1 Requirements

Functional specification documents signed off at the beginning

Traditional software methodologies need to gather the client’s requirements at the beginning of the project, putting everything in a big functional specification document that needs to be signed off by the client before it can go into production. This approach seems fair and quite straightforward but practice over the years has shown that it can have some very serious flaws. The biggest problem with this approach is that due to the problems in understanding software and how it works, the customer cannot always tell what he wants from the beginning, and might miss some very important aspects when the functional specification document is signed off.

There have been numerous cases in the past when projects were developed within budget, delivered on time and as asked for, by the functional specification document, but when put into production they were completely useless for the customer. Yes, the customer could be blamed for not knowing what he wants when he needs to, but the pure fact is that he is unhappy about paying for unusable software. A customer that had such an experience will be very reluctant when needing to sign off another functional specification document and in many cases a lot of overhead will be generated in development because of his fear of being forced to pay for unusable software.

Although big functional specification documents seem very good to the untrained eye, they do have serious flaws:

- They are very costly to build (how much time do you need to write 100-600 pages of text?)

- They might miss important aspects of the program. Written text is not very good at communicating things. The writer cannot capture whatever he needs to capture and the one that reads tends to imagine the rest. Imagination is good, but not in this case because, in this case, it is very costly.

- They tend to push the customer proxy. The more indirection and steps trough which something goes from the customer to the actual developer building the software, the more misunderstandings can, and usually do occur. When the information needs several nodes to be transmitted from the ones that know the business to the one that builds the program, important time is lost, and important precision.

- Big documents are hard to read. Someone might say it is important to capture everything there otherwise programmers would complain about not being complete, but over completing it is just as bad.

- What if something changes? What if at the time the document was written there was one situation and it just changed? What do you do? Throw up the part or the entire document? If you wrote it in 1 month, and it cost you x, you have just lost a 10-20-30-...100% of x and need to pay again for the new one.

Big functional documents are written to replace trust between the parties involved. We need to write everything in the specification document and cast it in stone, so it can't change and we can't be held responsible by the customer later. Where there isn’t trust, there isn't good business. Lack of trust is usually compensated with enormous overhead in work and cost. But on the other hand, not everyone is to be trusted.

Agile requirements

Mike Cohn wrote in User Stories Applied []:

“I felt guilty throughout much of the mid-1990s.[...]

I knew that what we were doing worked. Yet I had this nagging feeling that if we'd write big, lengthy requirements documents we could be even more successful. After all, that was what was being written in the books and articles I was reading at the time. If the successful software development teams were writing glorious requirements documents then it seemed like we should do the same. But, we never had the time. Our projects were always too important and were needed too soon for us to delay them at the start.

Because we never had the time to write a beautiful, lengthy requirements document, we settled on a way of working in which we would talk with our users. Rather than writing things down, passing them back and forth, and negotiating while the clock ran out, we talked. We'd draw screen samples on paper, sometimes we'd prototype, often we'd code a little and then show the intended users what we'd coded. At least once a month we'd grab a representative set of users and show them exactly what had been coded. By staying close to our users and by showing them progress in small pieces, we had found away to be successful without the beautiful requirements documents.”

Agile methodologies have developed methods to capture requirements in such a way that the problems and risks associated with the methods presented above are very well addressed. Agile thinking works with the fact that requirements change in time and works with human nature, allowing the customer not to know exactly what he wants from the beginning. In every agile methodology, a very strong emphasis is put on close and direct communication with the customer, so the misunderstandings generated by customer proxies are eliminated. The most efficient methods of communication are used: verbal, real-time, face to face communication is pushed forward against written documentation, which is kept to a minimum.

Intermediate work products (documentation, use cases, UML diagrams, database schemas etc) are not important models of reality, nor do they have intrinsic value. They have value only as they help the team make a move in the game. Thus, there is no idea to measure intermediate work products for completeness or perfection. An intermediate work product is to be measured for sufficiency: Is it sufficient to remind or inspire the involved group?

[Alistair Cockburn, 2003]

User stories, backlog items, acceptance tests

Since the emphasis is put on close collaboration and face to face, real time communication with the customer, written requirements are kept as brief as possible. They usually act mostly like a reminder of a thing that needs to be discussed between the customer and the developers.

The most well known ways to capture requirements from the customer in agile methodologies are the user stories, originating from XP and the backlog items, originating from SCRUM. Both describe functionality that will be valuable to the customer and the users of the system.

Mike Cohn defines in ‘User Stories Applied’ [], the following characteristics of a good user story: independent, negotiable, valuable to users or customers, estimable, small and testable.

Because of their light nature, user stories and backlog items act more like an instrument used for planning, than for describing what is wanted. They give the directives on what is wanted and the details are mostly discussed with the customer and expressed as acceptance tests.

Acceptance tests are a feedback mechanism defined by the customers (with the help of programmers) to allow programmers to know if a feature has been finished and if that feature has been implemented exactly as the customer wants it. Instead of making the customer describe what he wants, acceptance tests are built by asking the customer: “how will I know that what we did is according to what you want?”. And he will start saying what needs to be done, for him to be able to know if what was requested was built and finished or not. Instead of heavy descriptions, two purposes are achieved by this question: you know what needs to be done and you know how you can test, at the same time. Also, ambiguity tends to be dissolved by acceptance tests, because the customer, in order to know how to test something, he must have a very intimate knowledge and understanding of that thing.

The user story mini-sample

Case 1: Requirements document

4.3 Outstanding contracts report

The system will allow the users to see a report of outstanding contracts. An outstanding contract is a contract that exceeds a predefined number of days, since it has been sent to the customer and no response has been received.

The outstanding contacts page:

Contract management system My contracts (1) Reports (2) Outstanding contract report Choose the number of days: [ (3)] [Search] (4) Results: Customer (5) Contract reference (6) Number of days passed (7) Customer #1 Ref #1 3 Customer #2 Ref #2 0 Customer #3 Ref #3 2

#	Element	Description
1	My contracts	Description: Menu item (-the) will allows navigation to the page where my contracts can be managed
2	Reports	Description: Allows user to navigate to the reports screen where all reports reside
3	Number of days	Description: Allows entry of the number of days that need to be exceeded so that the contract will be displayed. Function: User types the number of days as an integer value
4	Search button	Description: Allows the query to be performed Function: On click the system will search the database for the contracts specified and will display them in the table below the button on finish. If no results have been found then a message “No search results.” will be displayed
5	Customer	Description: Displays the customer name
6	Contract Reference	Description: Displays the contract reference number
7	Number of days passed	Description: Displays the number of days that have passed since the contract was sent to the customer

Case 2: User story and acceptance test

Outstanding contracts report

Report all contracts sent to the customer and no answer was received for a number of days above a given amount.

Acceptance test

For a list of contracts:

contract	customer	Sent date	status
NL001	IBM	2006.04.01	Sent
RL001	Coca Cola	-	Preparation
DF001	HP	2006.04.05	Accepted
NL002	BT	2006.04.09	Sent

Test case 1: Filtering at 2005.10.20, for contracts above 100 days would result in:

No results to display.

Test case 2: Filtering at 2006.04.20, for contracts above 15 days would result in:

Customer	Contract reference	No of days passed
IBM	NL001	19

Test case 3: At 2006.04.30 for 10 days:

Customer	Contract reference	No of days passed
IBM	NL001	19
BT	NL002	11

3.2.2 Set-based development

Communicating the requirements can be done in two ways: communicating the constraints, allowing multiple solutions to be developed by the programmers until one emerges or communicates the solution, and refining it until it works. Mary Poppendieck calls this: set based vs. point based development.

As a simple example, the customer can say: we need a desktop application that enables us to see our sales or we need a way to see our sales. The first thing enforces a desktop application solution, while the second allows the possibility to develop more solutions: web based, desktop based, mobile etc. If the second requirement is detailed as: we need to be able to see our orders real-time, as quickly as possible as some of them need a sales manager’s approval, and he needs to do that no matter where he is, then suddenly, the mobile solution might seem like a better solution, having the mobility constraint. Further defining the constraints, one solution will emerge. If we take the point based approach going directly on a desktop application, when the mobility constraints is revealed we might be forced to go back and rethink the desktop solution, throwing away the desktop solution code.

Agile methodologies have built in their practices ways to communicate and implement, using set based development like: developing multiple solutions in the early iterations and letting one emerge, communicating constraints instead of solutions , a very good example being acceptance tests which communicate testing constraints not solutions for implementation and design for change, where extensive refactorings, and fast changes and shifts based on constraints can let a solution emerge from a larger range of possibilities.

3.2.3 Feedback

Customer feedback

It's two in the morning and you are driving home. The traffic light is red, and there's not another car in sight. But the traffic light is red, so you stop. And wait. And wait. Finally, the light changes, after allowing time for lots of nonexistent cross-traffic. You think to yourself, it’s going to be a long drive home. And sure enough, the next light is also red. But as you approach the light, it turns green. Ah ha! you think, An automatic sensor. That light is smart enough to know I'm here and there's no one else around. I hope the rest of the lights are like that!

The difference between the two lights is feedback. The first light was preprogrammed.

[Poppendieck 2003]

Improvement can only be done as a result of feedback. The development team writes a piece of the system, the customer sees it and tells them if that program is good or not. He can also tell the developers what would be even more liable to improve the program’s quality.

The virtual nature of software makes it very hard for customers to understand how it’s build and done, so it is very hard for them to communicate you what they really want. To address this extremely important problem, software should be shown very often to the customer. This has two very important benefits:

- he can see where you are, the real progress, and this increases his trust that you’re doing what you were planning to do and that his money is not wasted

- he can see if something is not what he wanted and tell the development team, so they can fix it or after seeing it he can start to ‘feel’ the software and be able to give you much better information about what he wants. The smaller the delivery cycles, the better.

The small delivery cycles are incredibly important in a project’s communication. As previously stated they show one very good technique to avoid derailing the project and they embrace changing requirements.

When the first requirements are being gathered from the customer, the developers explain him how something can be done, but it is very easy for him to imagine a completely different thing. It is also very hard for him to understand what can and what cannot be done; why something costs more or what costs less. After the team delivers the first result of an iteration, he can see a computer program, and everything seems to take shape in his mind. He can use this as a basis for future requirements, and can see about how much time one thing or another takes to be implemented. The second iteration is even better and as the project progresses, the dark fog that the software seemed to be in the beginning is starting to become clearer, easier to control and he becomes less and less stressed. He starts to understand how things are done and he sees real progress. There is nothing more honest than showing the customer the real program from which he can see the progress.

Code feedback

If feedback is the base for all improvement, how can feedback help the internal quality of a project? What we discussed earlier about feedback is the external quality that can be improved, the part that the customer understands and helps improving, but what about internal quality. But what is internal quality in a software project?

A project is of high internal quality if it can be tested and debugged easily and it can change with very little effort. Being able to be changed easily assures the program a much longer life in production, which makes it cheaper. If the customer pays $1.000.000 for a software and the software is used for 5 years, it will cost $200.000 every year. If the same software can be fixed, readapted and maintained longer in usage, for instance for 10 years, it costs $100.000 per year. Twice cheaper, isn’t it?

But how can we make the software give us feedback about its health? In agile methodologies the best technique for receiving feedback from the code itself about its health are automated tests. They can detect problems very early and report them to the developers so they can act very fast on them. The automated tests become break detectors. If the developers can detect breaks in the existing functionality fast, they can fix them fast and that allows them a great flexibility. A programmer makes a change in the software, even in the code he didn’t write. Then he runs the tests. If they detect a break, he knows it and he can choose either to reverse the change or repair the affected part.

A big problem in a program’s flexibility is that if it becomes big, it is very rigid, and making a change can affect parts of it that cannot be easily predicted. So the developers are afraid to change it. So the project starts to be doomed. The hardest it is to be changed, the hardest it is for it to cope with the market requirements so the shorter its life will be. Having an army of break detectors as automated tests ensure great flexibility, diminishes fear of change, even in big and complex systems, because of the fact that they can give the developers feedback about their actions very fast.

3.2.4 Showing progress

Working software doesn’t lie. One of the agile manifesto principles states: “Working software is the primary measure of progress”. Showing the customer the product instead of reports after each iteration is completed (2-4 weeks), is a great way to communicate progress and to build a product of much bigger value to the customer. Putting that software into production every few months and seeing how it is used, collecting what the users think can improve the software, can build thrust between the client and the programmers, driving the software to success.

For communication inside the team or with upper management, big visible charts on a wall, showing very clearly the features implemented, and the features not implemented can be a very effective and silent technique to show everyone passing by that wall, the current status of a project.

3.3 Communicating inside the team

The project related information must go very fast inside the team. Crystal calls this osmotic communication and it is at the very core of the methodology. It says that people working on the same project need to be able to use the best communication technique: face to face communication, with and without a whiteboard. For this the best technique is to have the entire team collocated in the same room as much as possible , all facing each other, to be able to ask each other questions and to enable others to hear discussions etc. The team must also have a whiteboard in their proximity where they can design or use sketches on the whiteboard to communicate better.

Extreme Programming goes even further, addressing the communication problem mainly with user stories written on pieces of paper and stuck to the wall, and with writing all the business code by pair programming. This means that two programmers, work together at the same computer, understanding 100% of what’s done, and not 50% as if they were doing it separately.

The techniques above and others mainly address communication when time is crucial, but there are other problems of communication within the organization, where time is not that crucial. This is best described by a situation, where a team is developing a program for a few months, delivering it to the client in the end. Now the client wants to continue the development of the program, to add extensions to it, to improve certain parts etc, but the team that built it is not available anymore or it is partially available. How will the new programmers who come in contact with the program understand what was required, what was done, and how it was done?

Extreme Programming comes up and says that the best way to document a computer program is the code itself and the automated tests, it has and not documents describing what was done, as these documents need to be written by someone, which is costly and becomes obsolete very soon. In most cases, I find this approach very good, as I noticed that, recently, whenever I download an open source framework or look at a code written in the past, the first thing I instinctually do when I want to learn how to use it, is to look at the unit tests. They show me how the software can be used, and which the limitations are, in the same language I want to use it: code.

3.4 Improving communication: quantity vs. quality

If communication plays such an important role in all aspects then how can we improve it? For the sake of improving communication, many people fall into a very big trap, believing that improving communication means increasing quantity: more meetings, more documentation, more emails and more phone calls. This is profoundly wrong: communication can only be improved by quality and in many cases improving quantity means increasing the communication problem and not solving it.

Meetings are failures when participants leave the meeting feeling no productive results were achieved; or, that the results that were derived were accomplished without the full engagement of the participants. In short, these useless meetings do not engage collaboration and participative decision making.

[ Booch and Brown, Collaborative Development Environments]

In order to improve the quality of communication we must see which types of communication are better then others. Following this idea, Alistair Cockburn, makes a communication top in Agile Software Development, concluding that Interactive, face-to-face communication is the cheapest and fastest channel for exchanging information.

If verbal, face to face communication is not at hand but there are other great ways to deal with it, like visual communication, web based collaboration tools acting as whiteboards and allowing people to chat real-time, application sharing, instant messengers, etc.

As an example of visual communication, a single screenshot can show even the context in which a bug appeared and give the developers a great deal of information about the user actions. Screenshots or even small recorded movies of user actions, are very cheap, and communicate a lot of useful information, which in some cases might be missed just by sending a plain text email or document.

Another example of improving communication are customer improvement programs where the user is either asked to add suggestions on how he would like a program to be improved or when a bug appears the software automatically detects it and sends by email or other means, information (maybe even screenshots) about the problem to the developers to be analyzed and the problem solved.

Macromedia Flash recorded movies are becoming a more and more popular way to show how software can be used instead of plain text or “classical” help documents, because their visual nature enhances the experience the user has when learning about the program.

Table based communication is one of the most used at this time in businesses (mostly due to Microsoft Excel) because of its simplicity and concreteness. The same approach has been brought into software development by Ward Cunningham, who invented a brilliant communication, organization and testing tool based on tables called Fit (http://fit.c2.com ). Fit is a great framework for acceptance tests. His work is now continued by a wiki concepts based tool developed at ObjectMentor, being supervised by Robert C. Martin, called Fitnesse (www.fitnesse.org ).

New ways of enhancing communication are emerging, allowing people to communicate better, thus making them able to collaborate closely even if they are not situated in the same room, and obtaining better software.