The problem of communication

1. Introduction

A lot of waste and rework is done because of bad communication. In fact, communication is the main factor in a project success or failure. And that does not refer only to software. Alistair Cockburn, wrote an entire book about the importance of communication and builds a methodology around improving communication: the crystal family

2. The big misunderstanding: improving means increasing quantity

The great misunderstanding: improving communication means increase quantity: more meetings, more documentation, more emails and more phone calls. This is profoundly wrong: Communication can only be improve by quality and in many cases improving quantity means increasing the communication problem and not solving it.

One of the most important principles in software is in my opinion looking back to what you did , so you can learn what was good and what was bad, and find ways to improve your activity. This is emphasized in the principles behind the agile manifesto as:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

In a project we did recently we looked back with the customer and we noticed that the bad things in a project developed were mostly caused by bad communication. By bad communication I mean much time needed to get questions answered, ambiguous requirements, not completely specified parts of the system, etc. The immediate answer was that we need to improve communication. For that the answer was that more emails will be sent, more meetings will be help, more conference calls will be held, the specification document will be even larger.

The immediate problem was overcommunication: we held meetings for the sake of having meetings, without a real need, which was in my opinion a great waste of valuable time. We wrote more emails, we held more conference calls, where we discussed about the weather or other pointless things and usually the discussions were very high management level, like are you confident with the plan, with statistics, with waste of time. There were gains, but unfortunately they were shadowed by the new problems.
I realized that people , no matter where they are from, treat the problem of communication not seriously enough. They just say we will do more, when more means quantity instead of quality. But quality is not an easy matter when it comes to communication.


Can communication be improved? What is quality in communication?

What is quality in communication? In seems that Alistair Cockburn discovered a way, based on his long and great experience, to measure communication quality. He says:

Interactive, face-to-face communication is the cheapest and fastest channel for exchanging information (see 1.)

Continuing with a communication top:



Following Alistair’s advice we need to speak more in front of a whiteboard. Why in front of a whiteboard? Because sometimes, a picture is worth a 1000 words, and this applies to software very well.

Going even further, I have found that a very practical way to improve the quality of communication is visual communication. I noticed that when describing bugs and ways to reproduce then, we or our clients have trouble describing then and describing a step by step process to reproduce them. For this one of the things I did was to ask that customers and others, when reporting bugs even changes, enhancements to include screenshots.

A single screenshot can show even the context in which the bug appeared and give you a great deal of information about the user actions. Screenshots are very cheap, and communicate a lot of information. They can even be automated into the software, so whenever you have a bug, the software takes a screenshot and sends it to you by email along with the last x lines in the log.

Many times, I am just lazy and I don’t want to read. Joel (www.joelonsoftware.com) says in his book: (Users Don't Read the Manual). Well that is very true. I want to use a new technology but I am too lazy to read the help files or the documentation. But wait, a new way of improving communication, emerged: small flash movies, showing you , visually what you need. They seem to pop everywhere. Just take a look at Ruby On Rails: [link] This can be done for bugs, for help, for a lot of things. And they are easier to do and send then writing a step by step document.

3. Communication divided by direction.

If we consider ourselves and the team the center of the communication, and we consider where the source and target of communication is, we can divide communication in:

- Communication within organization, team etc
- Communication outside with other stakeholders


3.1 Communication inside the team and organization

Depending on what needs to be communicated, the agile methodologies have ways to deal with communication inside the organization. In my opinion communication inside the team can be divided into two categories: project related and team performance related.

3.1.1 Project related

The project related information must go very fast inside the team. Crystal calls this osmotic communication and it is at the very core of the methodology. It says that people working on the same project need to be able to use the best communication technique: face to face communication , with and without a whiteboard. For this the best technique is to have the entire team collocated, in the same room as much as possible , all facing each other, to be able to ask each other questions and to enable others to hear discussions etc. The team must also have a whiteboard close.

Extreme Programming goes even further, addressing the communication problem mainly with user stories written on pieces of paper and stuck to the wall, and with writing all the business code by pair programming. This means that two programmers, work together at the same computer, understanding 100% of what’s done, and not 50% as if they were doing it separately.

The techniques above and others mainly address communication when time is crucial, but there are other problems of communication within the organization, where time is not that crucial. This is best described by a situation, where you have a team developing a program for a few months, delivering it to the client. Now the client wants to continue the program, add extensions to it etc, but the team that built it is not available anymore or it is available partially. How will the new programmers that come in contact with the program understand what was required, what was done, and how it was done.

Extreme Programming comes up and says that the best way to document a computer program is the code itself and the automated tests, it has and not documents describing what was done, as these documents need to be written by someone who is costly and become obsolete very soon. In most cases, I find this approach very good, as I noticed that recently whenever I download an open source framework or look at code written in the past, the first thing I instinctually do when I want to learn how to use it, I look at the unit tests. They show me how the software can be used, and which are the limitations, in the same language I want to use it: code.

3.1.2 Team performance related

We all want to improve our skills. Many times, we don’t have the information or the time to explore it. A learning environment has been clearly specified as a way to keep programmers motivated on the long term. In fact, this is one of the 7 principles of lean software development (see 3)

It has been proven that pair programming can help a less experienced programmer that is paired with a more experienced one move very fast ahead. The same result is achieved when the team is collocated and the less experienced can hear and see at the whiteboard the more experienced doing something, being able to ask and be given answers very fast.

One great technique, described by Alistair Cockburn is information radiators. These are things you would like to be communicated to others. These can be of help both in project, as you might have the stories stuck on the wall, divided in iterations and in sections: done, in progress and not done. Whomever goes by can see which are done, which are not, very easily without the need of asking anyone about the progress. Ron Jeffries wrote a great article about how to use the walls to communicate about progress, called Big Visible Charts. (see 4)

The information radiator are usually characterized by being pieces of paper with information that are put in places that readers might find them and read them. Usually, after reading a book, in order to get the others to read it, I cut some of the paragraphs I liked most, print it on pieces of paper and put it all over the working environment: on the walls, where people take breaks and smoke cigarettes, in places they feel comfortable. Usually after sticking them on the walls I hear the others referring to them in our conversations. It is also a very good way to speak to managers, with notes stuck on the walls.

Another technique, I found very valuable, to get people to move ahead, improve fast is what I call school like meetings. I have described them in : [Self training … ]. This is mainly , an hour where we take a subject and all read it, then based on some questions we discuss it so we can make sure everyone understands it. This is done within working hours, usually at the first hour in the morning. We have read even a book, chapter by chapter in this manner. It has proven an enormously healthy practice. Let me give you an example.

We started using an ORM framework. I started using it in a project where I teamed up with another developer. I had more experience with ORMs and I showed him a few tricks. At some time, I moved to another project and he was on his own. Then a new project was started and the ORM was used, then another all by other people. After about 4 months I looked into the code of one and I was very badly surprised that is was completely misused. People even were complaining about using this ORM because it is not of any real help. I decided to go to the bottom of the problem and I noticed that the second project was started, having the first project as an example. The mistakes from the first project that my colleague did were moved to the second and taken from granted. The third even more. A lot of people were using the ORM, but the level of knowledge after 4 months was very low.

I had to do something. I immediately gathered everyone, in that afternoon, took the documentation of the ORM and we stopped work and everyone had to read the documentation. I established a set of questions, so I can discover if the technology was reasonably well understood and we started reading. To read, answer the questions it took 4 hours. It took 4 hours to get everyone to understand how this technology. Four hours of no work, a time that could be used to build software. But those for hours proved later one of the best well spent hours in our activity. By understanding the technology in 4 hour they avoided hundred of hours of misusing it and building code that was just reinvented or badly written. Countless hours of debugging were avoided by understanding the ORM and using it right, which decreased the quantity if code by up to 75% and by having a smaller code, less bugs were produced and debugging was a lot easier.

3.2 Communication outside with other stakeholders

The usual stakeholders in a software project are: the customer, the management and the programmers. The management and the programmers can be considered either inside or outside the same circle, so both techniques are applicable. One of the very effective, silent techniques of communicating with management inside is by using the big visible charts that Ron Jeffries describes. Management can be considered depending on specific contexts both inside the programming environment and outside.

Communicating outside the programming environment is divided in my opinion in 2 sections:

1. From outside the team to programmers
2. To outside by programmers

The first can be further divided between:

1. From outside at the beginning, when code is to be written according to what is communicated
2. From outside at the end or after some code is delivered, commonly known as feedback

The software industry seems to consider that the first forms of communication from outside, about what needs to be coded are the requirements document or functional specifications.

Are functional requirements a good way to tell programmers what to build?

In a project we did before, we joined another company to develop it, having developers from the two companies developing the project. With the other company not being an agile oriented company, but more like waterfall process oriented, they came to us with a big functional specification of more then 100 pages. This was later estimated by us to be included in the first 3 week iteration.

The interesting part is that usually when people see such a nicely layout with so much information about screens, user interaction , use cases, their first thought is that this is such a good document with all the information captured and put where it belonged. Everyone looks up to those that did it. It makes some people just say: now this is how things get done properly. But are they?

My first thought was, if we develop this part of the software in 3 weeks, how many weeks was the functional spec written on? I just hoped it was clear so it does not need even more time to be invested in it. Another reaction I saw in my colleagues was that because the document was so big the tended to overlook some of the content. It was too much text. What a mistake that was, because we missed some well hidden big requirements, that were missed in the estimate, but were required at the end of the iteration.

So, now the conclusion: although big functional specification documents seem very good to the untrained eye:
- They are very costly to build (how much time do you need to write 100 pages of text?),
- They might miss important aspects of the program. As Alistair already pointed out, they are not very good at communicating things. They are indirection. The writer cannot capture whatever he needs to capture and the one that reads tends to imagine the rest. Imagination is good, but not in this case because in this case is very costly. This is a problem with alternative means of capturing requirements like direct talk, stories and acceptance tests but the main difference is the amount of time needed to spot those misunderstandings.
- They tend to push the customer proxy, which is the root of all evil. The more indirection, the more misunderstandings. When the information needs several nodes to be transmitted from the ones that knows the business to the one that builds the program, important time is lost, and important precision.
- Big documents are hard to read. Someone might say it is important to capture everything there otherwise programmers would complain about not being complete, but over completing it is just as bad
- What if something changes? What if at the time the doc was written there was one situation and it just changed? What do you do? Throw up the part or the entire document? If you wrote it in 1 month, and it cost you x, you have just lost a 10-20-30-...100% of x and need to pay again for the new one.

My belief is that big functional documents are written to replace trust between the parties involved. We need to write everything in the spec and cast it in stone, so it can't change and we can't be attacked by the customer later. My belief is that where there isn't trust, there isn't good business. Lack of trust is usually compensated with enormous overhead and cost. But after all, not everyone is to be trusted.

Although they are not evil, I am trying to show that big requirements documents are just not the answer in most projects, anymore. We need to move faster and for that the communication needs to be efficient and light. All agile methodologies say that you should have the customer (or customer representative) with you while developing. That enables the most efficient way to communicate. Direct, face to face talk, in front of the running program and in front of the whiteboard. But in order to be organized and not to replace big heavy, beaurecratic methods to communicate to chaos, we still need to have something written so we can organize, plan light. This text that is being written needs only to remind us of a thing that the customer needs implemented. It needs to be able to be written fast, so in case changes come up can be thrown away with no remorse, can be estimated and have business value. This is what is known as a user story.

In the wonderful book ‘User Stories Applied’ of Mike Cohn he defines these characteristics of a user story:
- Independent
- Negotiable
- Valuable to users or customers
- Estimable
- Small
- Testable

To demonstrate a way of improving quality vs quantity in communicating what needs to be done, agile methodologies (re)invented acceptance tests. Instead of making the customer describe what he wants you ask him: “how will I know that what I did is according to what you want?”. And he will start saying what needs to be done to be able to know if what was requested was build and finished or not. Instead of heavy descriptions, two purposes are achieved: you know how you can test and what needs to be done at the same time. Also, ambiguity tends to be dissolved by acceptance tests, because a man in order to know how to test something he must have a very intimate knowledge and understanding of that thing.

Feedback

One of the Extreme Programming values is feedback. If the customer knows how to specify how you can test that you’re done building what he requested, after delivering how can you really be sure? You need his input after seeing and testing the program.

Improvement can only be done as a result of feedback. You write a program, the customer sees it and tells you if that program is good or not. He can also tell you what would be even better to improve the program’s quality.

The virtual nature of software makes it very hard for customers to understand how it’s build and done, so it is very hard for them to communicate you what they really want. To address this extremely important problem, software should be shown very often to the customer. This has two very important benefits:
- he can see where you are, the real progress, and this increases his trust that you’re doing what you were planning to do and that his money is not wasted
- he can see if something is not what he wanted and tell you, so you can fix it or after seeing it he can start to ‘feel’ the software and be able to give you much better information about what he wants. The smaller the delivery cycles, the better.

The small delivery cycles are incredibly important in a project’s communication. They are called within agile communities, iterations and it has been proven that it is best to be between one and 4 weeks in length. As previously stated they give you one very good technique to avoid derailing the project and they embrace changing requirements.

In the past there have been projects, in which we worked for 4-6-8 months and then one day, it was the big day: the software was presented to the client. That day was one of the most stressful situations I and the other programmers could be in. It feels just like waiting in the court of law for the verdict to be pronounced. And the stress is as big as the risk is. What if the customer says that that is not what he wanted? The losses for your company’s 4 months of development are big, and redoing the software is something any business can very hardly cope with. With iterative development, showing the customer the result of the latest iteration, can mean in the worst scenario, that 2 weeks of work are useless. But 2 weeks is much easier to solve then 4 months or more. It is much easier to put one wagon on the rails again, then the whole train.

When the first requirements are being gathered from the customer, you explain him how something can be done, but it is very easy for him to imagine a completely different thing. It is also very hard for him to understand what can be and what cannot be done. What costs more or what costs less. After you deliver the first result of an iteration, he can see a computer program, and everything seems to take shape in his mind. He can use this as a basis for future requirements, and can see about how much time one thing or another takes to be implemented. The second iteration is even better and as the project progresses, the dark foggy unknown that software seemed for him in the beginning is stating to become more easy to control, more clear and he becomes less and less stressed. He starts to understand how things are done and he sees real progress. There is nothing more honest then showing the customer the real program from which he can see the progress.

If feedback is the base for all improvement, how about the internal quality of a project. What we discussed earlier about feedback is the external quality that can be improved, the part that the customer understands and helps improving, but what about internal quality. But what is internal quality in a software project?

A project in my opinion is of high internal quality if it can be tested and debugged easily and it can changed with very few effort. Being able to be changed easily assures the program a much longer life in production, which makes it more cheaper. If the customer pays $1.000.000 for a software and the software is used for 5 years, it will cost $200.000 every year. If the same software can be fixed, readapted and maintained longer in usage, for instance for 10 years, it costs $100.000 per year. Twice cheaper. Well, customers love that.

But how can you make the software give you feedback about its health? In agile methodologies the best technique for receiving feedback from the code itself about its health are automated tests. They can detect problems very early and report them to you so you can act very fast on them. I call my automated tests, break detectors. They help me by giving me fast feedback about things that broke in my software. If I can detect them fast, I can fix them fast and that allows me a great flexibility. I make a change in the software, even in code I didn’t write. Then I run the tests. If they detect a break, I know and I can choose either to reverse the change or repair the affected part. A big problem in a program’s flexibility is that if it becomes big, it is very rigid, and making a change can affect parts of it that you cannot know of. So you are afraid to change it. So the project starts to be doomed. The hardest it is to be changed, the hardest it is for it to cope with the market requirements so the shorter its life will be.

If I were to make a comparison, I would say that If I were the general and my code the army fighting the requirements, my automated tests would be my intelligence center. The bigger my army would get, and the larger the war zone, the hardest it would be for me to know what’s happening in all areas and be able to adapt my strategy to the new conditions. But If I have the break agents placed with the different parts of the army they can report to me about problems and defeat situations very fast. The more agents, the largest the areas of coverage, from where I can receive information from and act upon it.

Feedback is the one thing that corrects and improves the software.

Communicating with the outside

Before starting a project, teach your customer about the importance of good communication and how that is achieved in software and with you in particular. Show a very honest plan, of how the project will be build.

On misunderstanding with agile methodologies, is that it is believed that it is not well planned and to the uninformed this seems like cowboy programming, where everyone takes arbitrary decisions and wait to see what happens. In fact, agile software methodologies are very well planned. They plan light, having the possibility of change in their minds, making the plans simple and very easy to change. The plans are built together with the customers, management and the programmers, on very honest and proved facts, both at a high level, called release planning and at low level called iteration planning.

Projects in Extreme Programming for instance, are divided into releases, which are to be delivered to the real users, or that are going ‘live’ and usually take one to four months to develop and they are also divided into iterations that are between one and four weeks in length and which need to deliver a tested, production ready system to the customer that is in charge of developing the software.

When making the big, release plans, the customer tells the programmers about what he wants as small sentences written on small paper cards, called user stories. All these are gathered from the customer and then estimated roughly by the customer. After being estimated, the customer can see their cost, and then be able to prioritize them. After the prioritization is done, the user stories are put in the iterations, to be developed. Well, this isn’t very different from traditional planning, excepting the level of detail of the user stories. But wait, I forgot to tell you that the plans can be updated after each iteration. This gives great flexibility in what is being built, and it also gives the customer great control on how and what is being built. He is in charge. After all, if you planned to release the software after three months with the customer, and after one month he realized that he misspecified what he wanted, or the business conditions changed, he can turn the direction very easily to the new one. This gives agile project great adaptability which can deliver the customer a much better product, as it can be easily changed and the control is in his hands.

After each iteration, the customer sees the software planned and developed in that iteration, and can readapt according to what he sees. He can reprioritize, give up some features and add new ones. If the plan is light, it can be easily changed and readapted. He receives the best feedback about what you’re doing for him by seeing the software and this gives him the possibilities to tell you how to improve it, by giving you feedback.

Showing the customer, the product instead of reports, in the middle of the production, is a great way to communicate and build a product of much bigger value to the customer.

Conclusion

Although, maybe what I have said earlier might need a level of knowledge about agile technologies, I hope it can also open an interest in those that have not used them , by showing different strategies on how the problem of communication is addressed by agile methodologies. This really can improve products, customer satisfaction, can decrease costs and in all improve the way we do business.

Charging the customer for miscommunication, is the worst way to address the communication problem, however it is still extensively used. This makes software more expensive, and generates new problems and new stresses that can be pushed to programmers or customer. Just as a simple example, if it costed me x+something to build the detailed requirements documents, and the customer doesn't want to pay more then I'll just get the programmers to build the software in less time with less people. Guess how much programmers under stress cost?

1. Agile Software Development, Alistair CockBurn
2. User Interface Design for Programmers, Joel Spolsky
3. Lean Software Development: An Agile Toolkit, Mary and Tom Poppendiek
4. Big Visible Charts, Ron Jeffries