Friday, December 19, 2014

Functional programming (in javascript and python): Part 1: Why?

The problem

At the beginning of this year I started thinking how would I describe what I did as a programmer for the last 13 years in just a few words, and this is the conclusion:

List processing and debugging. 

Now let me explain: If I abstract enough, everything looks like a list: arrays, dictionaries even objects, considering them a list of properties with values. Complex objects become lists of lists. They're everywhere: from databases where these days you get back lists of rows/objects, to models, to the UI for instance $("...") in jQuery. So basically all day long I wrote functions and methods to process these lists and lists of lists, transforming them, changing them, persisting and loading them, send and receive them through the networks and internet or displaying them on the screen.

The big problem is that if the code becomes a little complex, you expect at the end of some processing some result and it's not there. Basically you expect the system to be in a state, and it is in another, thus having an error or a bug, so you start following what happens to see why. If the system uses events and async operations your debugging work starts to take a lot of time and effort. Thinking about time, I would correct the conclusion above to

30% list processing, 70% debugging

The causes

For years I looked for solutions: from OOP, to patterns, to separation of concerns, agile methodologies practices such as unit testing and test driven development. All helped, but none seem to address the biggest issues: state is everywhere thus unexpected state changes, not to mention concurrent programming.

The solution

Then I remembered about LISP (List programming), and how strange it seemed. But somehow it seems to do exactly what I needed, unfortunately not in the programming languages that I needed, but then I started to realise that the languages I worked with are multi paradigm and yes functional programming concepts are embedded right in (javascript/python).

So basically what functional programming addresses is: list processing using 3 main functions: map, reduce and filter. Using these 3 functions, simpler data structures like arrays, dictionaries and lists instead of objects used as data structures and higher order functions, the code to process lists becomes simpler and smaller.

Example

Let's see the total of an order in the shopping cart, for groceries:

Python, using imperative programming and OOP (badly, like pretty much everyone):

class Order()
     orderItems = []
     def __init__():
           pass
...

class OrderItem()
     name = None
     quantity = 0.0
     price = 0.0

     def __init__(name=None, quantity=0.0, price=0.0):
           self.name=name
           self.quantity =quantity
           self.price=price
...

order = Order()
order.orderItems.append(OrderItem(name="", quantity=1,price=0.99))
order.orderItems.append(OrderItem(name="", quantity=1,price=0.99))

def is_grocery(item)
     return ...

sum = 0
for item in order.orderItems
     if is_grocery(item):
         sum+=item.quantity*item.price

print sum

Python, using FP:

order = {"orderItems":[
          { "name":"sada""quantity":2,"price":0.99},
          {"name":"zjdfj""quantity":1,"price":2.99},
          ...]}

def is_grocery(item):
      return ...

print reduce(
        lambda total1,total2: total1+total2,
        map(
            lambda orderItem: i1["quantity"]*i1["price"]
            filter(
                 lambda orderItem:is_grocery(orderItem),
                 order["orderItems"]
            )
        )

About state changes, we'll speak in the next article.

Conclusion

So why didn't functional programming replace imperative programming since it has many areas where it clearly is better (maybe not in the example above)? Because it is much more difficult to understand and use, even though the resulting code is smaller. If you look at the code above you see that is starts the other way round: it wants to "reduce" 2 totals for order items and then you see that the reduce is applied on a list of totals for order items which is applied on a filtered list of items that are groceries. This is the reason people say functional programming shows what the code does not how.




Labels: , , , ,

Thursday, July 31, 2014

Hierarchical persistent data structures in javascript (using underscore.js)

Lately, I have been studying a lot about functional programming, and things like LISP, Clojure, Scala but since mostly I program in Javascript and python, I also studied a lot the functional programming concepts in these 2 languages.

The problem

I have a model (using MVC pattern) and I want to keep both the initial model state, and the one that is changed afar some operations. I wanted initially to use something like cloning or deep copying but it seems just to waste memory, if for instance I change only one thing in a hierarchy, so I don't need everything clones.

The solution : persistent data structures

I knew that what I wanted is to use persistent data structures in javascript and although they are available in libraries such as immutable or mori, I needed one that is hierarchical.

Persistent data sturctures acording to wikipedia:

In computing, a persistent data structure is a data structure that always preserves the previous version of itself when it is modified. Such data structures are effectively immutable, as their operations do not (visibly) update the structure in-place, but instead always yield a new updated structure. (A persistent data structure is not a data structure committed to persistent storage, such as a disk; this is a different and unrelated sense of the word "persistent.")

Solution:


So I want something like:

  initial_state = {a:1, b:2};  
  modified_state = change(model,"a","abc"))   

where initial_state remains unchanged and modified_state is {a:"abc",b:2}. This is very simple using something like _.clone(initial_state) in underscore.js. But what if I want something like:

  initial_state = model = {l:[{id:1}, {id:2}], b:2};  
  modified_state = change(model,"l[1].id",3)  

resulting in: modified_state = {l:{id:1},{id:3}], b:2}. Here copy, deep copy or clone doesn't help. So what if I write a function like this:

 function change(obj,path,val)  
 {  
   var new_obj = _.clone(obj);  
   var path = path.replace("[",".").replace("]","");  
   var levels = path.split(".");  
   if(levels.length>1)  
   {  
     var head = _.first(levels)  
     var tail = path.replace(head+".","")  
     new_obj[head] = change(obj[head],tail,val)  
   }  
   else  
   {  
     new_obj[path] = val;  
   }  
   return new_obj;  
 }  

It works!

I can even do:

 initial_state = {g:  
     [  
       {id:1, values:[1,2,3]},  
       {id:2, values:[3,4]}  
     ],  
   b:2}  
 new_state1 = change(model,"g[1].values[1]",7);  
 new_state2=change(model,"g[1].values",[7,2]);  











Wednesday, July 10, 2013

We were right: 3D is the future of UI/UX as iOS7 proves it

When we developed TaiaApps iPad app (www.taia76.com) we wanted to explore the possibility of building the UI, in a manner very little explored, but with massive potential for the future: 3D user interfaces. Apparently we were right guessing, what the next trend will be, as seen by the many 3D effects the new iOS7 has (parallax, movement of the folders, the new applications screen):


Thursday, March 28, 2013

10 greatest technologies of the last 13 years in my programming career

I was thinking today about the greatest programming languages, frameworks and technologies that really had an impact in my life and career as a programmer.

I will start from the beginning:

1. Java (In 2000 I had the choice of using C++ or Java, and I thought Java was incredible. In the meanwhile it might have come and gone)

2. JSP and Expression Language EL - Web programming with Servlets wasn't exactly easy

3. Hibernate - the ORM that showed it is possible

4. Ruby On Rails  - the MVC framework that simplified and let to better organised web application code forever

5. C# and .NET - The Monorail/Activerecord/Castle frameworks - like Java, they came and now they're obsolete, at least for me. But in 2005, boy they were great!

6. Apache Lucene  - searching for words in text would never be the same as before.

7. Javascript and jQuery - when I realised how powerful Javascript can be, I started to really doubt the power of static languages

8. HTML5/CSS3 - web would never be the same after this (CSS transforms and HTML5 canvas will change the world forever). And combined with PhoneGap led to TaiaApps out first iPad app, with a 3D UI ( www.taia76.com )

9. NoSQL databases - CouchDB - thinking about databases and how they keep data is now changed, as I said before - forever. Guess what powers TaiaApps?

10. Python - Flask/SQLAlchemy - no doubt Python is a great language, but SQLAlchemy is by far the greatest ORM I have ever used.


Saturday, December 29, 2012

TaiaApps 3D: discover cool apps recommended by your friends - concept

Friday, December 21, 2012

3D user interfaces: Navigating in space between information nodes

Each information node is displayed in the middle and the related ones are grouped around it, by relation type. Tapping another information node, takes you to it, and it brings it's own related content into the scene.

In our cases the information nodes can either be iPad apps or people that recommend them.






www.taia76.com

Thursday, December 20, 2012

3D user interfaces: lists and scrolling in 3D

In a 3D UI traditional vertical, or even touch horizontal scroll doesn't work anymore. Instead, the information is divided in pages, and they are stacked one in from of the other. When the user swipes the page it moves to become the last in the stack, and the one behind it becomes first. Swiping in the other direction reverses the action.







More on the app website:
www.taia76.com

Sunday, December 16, 2012

3D user interfaces in HTML5/JavaScript

I have always thought, that the next step in UI design, is adding the 3rd dimension. Throughout the history, there have been many attempts, but all of them failed because instead of simplifying the user experience they just made it harder.

Last night, I and a friend released our first app, for iPad, that is our approach on 3d user interfaces.


There have been many challenges, especially since the entire UI is HTML5/Javascript based. But one by one we managed to overcome them. How would you think scrolling in 3d works? :)

More information on the website: www.taia76.com

Monday, September 17, 2012

MVC in javascript

See code, running application and qunit test here: http://www.danbunea.ro/blogspot/mvcjs/

Problem

Today, in modern web applications, more and more code is moved from being generated server side (in java, c# with asp.net , ruby on ror, php) to be executed on the client browser, in javascript. This created a massive problem for all those used to having code well organized in modern web frameworks using the very powerfull Model View Controller pattern, since most of the code is written now in javascript.

Could MVC in javascript be a solution?

Well, there are more and more frameworks trying to adress the problem of having tons of javascript, whioch becomes a nightmare as well as replace the dynamic generation on the server , thinks such backbone.js, but I find it not very MVC. So why wouldn't I actually write my own MVC framework.

The first thing we want to do it actually list the users on the page, so that means the controller is invoked, it takes the decistion to load the users from the server into the model, and after that send the model to the view to render the users as html. So we have an html file index.html

...
<body>

<div id="list">
 <a href="http://www.blogger.com/blogger.g?blogID=9648422#" id="add" onclick="controller.add();">Add</a>
 <div id="usersList">
 </div>
</div>
<div id="form">
</div>
<div id="tests">
 <h1 id="qunit-header">
mvc.js</h1>
</div>
</body>  
    <h2 id="qunit-userAgent">
</h2>
<ol id="qunit-tests">  
    </ol>
</pre>
<script>


 


jQuery(window).load(function () {

    controller = new UsersController();
    controller.index();
});

</script>


</body>

</html>
 

now the controller class:

UsersController.prototype = new Object;
//constructor definition
UsersController.prototype.constructor = UsersController;


function UsersController() {
 this.model = new UsersModel();
 this.view=new UsersView();
};


UsersController.prototype.index=function(){
 this.model.loadUsers();
 this.view.renderUsersList(this.model);
};

and the model:

UsersModel.prototype = new Object;
//constructor definition
UsersModel.prototype.constructor = UsersModel;

function UsersModel()
{
 this.users=[];
}

UsersModel.prototype.loadUsers=function()
{
 this.users = eval(new HttpRequest().get("http://localhost/users/list.aspx"));
};
 
now let's see the view that renders the list. For it we actually use jQuery templates:

UsersView.prototype = new Object;
//constructor definition
UsersView.prototype.constructor = UsersView;

function UsersView()
{
    //templates
    this.userLine = "<div if='user${user.id}' class='user' >"+
 " <div id='fullName${user.id}' class='cell'>${user.fullName}</div>"+
 " <div id='username${user.id}' class='cell'>${user.username}</div>"+
 " <div id='email${user.id}' class='cell'>${user.email}</div>"+
 " <div class='cell'><a id='edit${user.id}' href='#' onClick='controller.edit(${user.id})'>Edit</a></div>"+
 " <div class='cell'><a id='delete${user.id}' href='#' onClick='controller.delete(${user.id})'>Delete</a></div>"+
 "</div>";
 jQuery.template("userLine", this.userLine);

}

UsersView.prototype.renderUsersList=function(model)
{
 jQuery("#usersList").html("");
 jQuery("#list").show();
 jQuery("#form").hide();

 for(var i=0;i<model.users.length;i++)
 {
  jQuery.tmpl("userLine", { user: model.users[i] }).appendTo("#usersList");
 }
};

Hmmm, very simple. Now let's implement the add:

UsersController.prototype.add=function(){
 this.view.renderAdd();
};

UsersController.prototype.save=function(id,fullName,username,password,email){
  id=this.model.users.length+1;
  this.model.users.push({id:id,fullName:fullName,username:username,password:password, email:email});

};
 
and in the view:

UsersView.prototype = new Object;
//constructor definition
UsersView.prototype.constructor = UsersView;

function UsersView()
{
    //templates
    this.userLine = "<div if='user${user.id}' class='user' >"+
 " <div id='fullName${user.id}' class='cell'>${user.fullName}</div>"+
 " <div id='username${user.id}' class='cell'>${user.username}</div>"+
 " <div id='email${user.id}' class='cell'>${user.email}</div>"+
 " <div class='cell'><a id='edit${user.id}' href='#' onClick='controller.edit(${user.id})'>Edit</a></div>"+
 " <div class='cell'><a id='delete${user.id}' href='#' onClick='controller.delete(${user.id})'>Delete</a></div>"+
 "</div>";
 jQuery.template("userLine", this.userLine);

    this.userForm = "<div class='fields'  style='display:table-row'>"+
 " <div id='fullName_${user.id}' style='display:table-row'>Full Name:<input id='fullName' type='text' value='${user.fullName}'/></div>"+
 " <div id='username_${user.id}' style='display:table-row'>Username:<input id='username' type='text' value='${user.username}'/></div>"+
 " <div id='password_${user.id}' style='display:table-row'>Password:<input id='password' type='password' value='${user.password}'/></div>"+
 " <div id='email_${user.id}' style='display:table-row'>Email:<input id='email' type='text' value='${user.email}'/></div>"+
 " <a id='save' href='#' onClick='controller.save(\"${user.id}\",jQuery(\"#fullName\").val(),jQuery(\"#username\").val(),jQuery(\"#password\").val(),jQuery(\"#email\").val());'>Save</a>"+
 " <a id='cancel' href='#' onClick='controller.list();'>Cancel</a>"+
 "</div>";
 jQuery.template("userForm", this.userForm);
}


...



UsersView.prototype.renderAdd=function()

{

    var user={fullName:"",username:"",password:"",email:"" };



    jQuery("#form").html("").show();

    jQuery("#list").hide();



    jQuery.tmpl("userForm", { user: user}).appendTo("#form");

};

Now what about edit?

UsersController.prototype.edit=function(id){
 var user = this.model.findUserById(id);
 this.view.renderEdit(user);
};

UsersController.prototype.save=function(id,fullName,username,password,email){
 if(id=="") 
 {
  id=this.model.users.length+1;
  this.model.users.push({id:id,fullName:fullName,username:username,password:password, email:email});
 }
 else
 {
  var user = this.model.findUserById(id);
  user.fullName=fullName;
  user.username=username;
  user.password=password;
  user.email=email;
 }
 this.view.renderUsersList(this.model);
};

and in the model:

UsersModel.prototype.findUserById=function(id)
{
 for(var i=0;i<this.users.length;i++)
 {
  if(this.users[i].id==id)
  return this.users[i];
 }
 return null;
};

and the view is refactored to:

UsersView.prototype = new Object;
//constructor definition
UsersView.prototype.constructor = UsersView;

function UsersView()
{
    //templates
    this.userLine = "<div if='user${user.id}' class='user' >"+
 " <div id='fullName${user.id}' class='cell'>${user.fullName}</div>"+
 " <div id='username${user.id}' class='cell'>${user.username}</div>"+
 " <div id='email${user.id}' class='cell'>${user.email}</div>"+
 " <div class='cell'><a id='edit${user.id}' href='#' onClick='controller.edit(${user.id})'>Edit</a></div>"+
 " <div class='cell'><a id='delete${user.id}' href='#' onClick='controller.delete(${user.id})'>Delete</a></div>"+
 "</div>";
 jQuery.template("userLine", this.userLine);

    this.userForm = "<div class='fields'  style='display:table-row'>"+
 " <div id='fullName_${user.id}' style='display:table-row'>Full Name:<input id='fullName' type='text' value='${user.fullName}'/></div>"+
 " <div id='username_${user.id}' style='display:table-row'>Username:<input id='username' type='text' value='${user.username}'/></div>"+
 " <div id='password_${user.id}' style='display:table-row'>Password:<input id='password' type='password' value='${user.password}'/></div>"+
 " <div id='email_${user.id}' style='display:table-row'>Email:<input id='email' type='text' value='${user.email}'/></div>"+
 " <a id='save' href='#' onClick='controller.save(\"${user.id}\",jQuery(\"#fullName\").val(),jQuery(\"#username\").val(),jQuery(\"#password\").val(),jQuery(\"#email\").val());'>Save</a>"+
 " <a id='cancel' href='#' onClick='controller.list();'>Cancel</a>"+
 "</div>";
 jQuery.template("userForm", this.userForm);
}

UsersView.prototype.renderUsersList=function(model)
{
 jQuery("#usersList").html("");
 jQuery("#list").show();
 jQuery("#form").hide();

 for(var i=0;i<model.users.length;i++)
 {
  jQuery.tmpl("userLine", { user: model.users[i] }).appendTo("#usersList");
 }
};

UsersView.prototype.renderAdd=function()
{
 var user={fullName:"",username:"",password:"",email:"" };
 this.renderForm(user);
};

UsersView.prototype.renderEdit=function(user)
{ 
 this.renderForm(user);
};


UsersView.prototype.renderForm=function(user)
{
 jQuery("#form").html("").show();
 jQuery("#list").hide();

 jQuery.tmpl("userForm", { user: user}).appendTo("#form");
};

Conclusion

As you can see above, implementing MVC in javascript can be very simple and doesn't require any sort of frameworks. Once MVC is implemented the code is well organized depending on concerns: who commands - controller, where by one look you can see what the entire code does (list, add, save, edit) the model who holds the data and exchanges it with the server and the view which actually renders it in html when asked by the controller.Simple to implement, well organized, easy to follow the code, very extensible.

Friday, September 14, 2012

TDD - scurta introducere practica (romanian only)

O scurta intoducere practica in Test Driven Development.

Download prezentare ca pdf
Demo unit test
Demo integration test

Articole despre TDD:
http://danbunea.blogspot.ro/2005/12/test-little-code-little-practical.html
http://danbunea.blogspot.ro/2007/01/test-first-web-applications-tdding.html
http://danbunea.blogspot.ro/2006/05/refactoring-legacy-web-application.html
http://danbunea.blogspot.ro/2005/12/tdding-sales-report.html
http://danbunea.blogspot.ro/2005/11/model-view-presenter-is-testing_27.html
http://danbunea.blogspot.ro/2007/03/testing-dragndrop-with-watin.html
http://danbunea.blogspot.ro/2009/05/why-bdd-can-it-help-me.html

http://danbunea.blogspot.ro/2008/05/chapter-6-quality-and-testing.html
http://danbunea.blogspot.ro/2005/09/how-tdd-improves-development-speed-and.html
http://danbunea.blogspot.ro/2006/06/to-obtain-good-code-writing-tests-and.html


Wednesday, January 04, 2012

7 deadly sins of automated software testing

Monday, October 31, 2011

Great advice on the programming career

Wednesday, May 06, 2009

Why BDD? Can it help me?

I've been watching a lot of people arround me and on the internet talking about BDD. Of course as alwars, some were more extremist and said that TDD is dead, and BDD is much better. 

Why is BDD needed, in general?

Most people develop code, and design, mainly on the simple tasks, by exploration. They write some code, see if it works, refactor it, make it better, then if it still doesn't work they sit down and think of a better solution. After this 3rd stage the real thinking is needed. This fact comes from human nature and the fact that most of us like to do things with the minimum effort required. Unfortunately, writing tests first, moves the real thinking phase ahead, thus making it more unconfortable, because the first thing you need to do is to think how to design the test that will design your code. This isn't as simple as just starting to type code then seeing if it works. So, conslusion one would be that writing tests first, is harder, because it doen't allow the people to warm up, before being sent onto the pitch. 

On the other hand, it seems that there are still lots of people that do not make a clear difference between writing automated tests and Test Driven Development, where the tests are written first, and actually drive the way you're developing the code, making it usually smaller, and less dependent, as a general rule better. This fact has been noticed by some incredibly smart people, and they said maybe TDD is too abstract, and we need a simpler way to show people the philosophy behind it, so we need to simplify it and enforce somehow people to write the tests first, and let the the tests drive the way the code is written. So the second concusion is that, maybe TDD should be simplified, renamed, handed out to the masses, as Dan North (original developer of BDD concept) said:

I had a problem. While using and teaching agile practices like test-driven development (TDD) on projects in different environments, I kept coming across the same confusion and misunderstandings. Programmers wanted to know where to start, what to test and what not to test, how much to test in one go, what to call their tests, and how to understand why a test fails.

Why do I need it?

After practicing TDD for many years, I saw one drawback, that kept coming to me: in time, I forgot what they did, and sometimes I needed lots of time to understand them again. Unfortunately for this there isn't a solution:

 - small easy to understand unit test: If you develop lots of small unit tests, lots means that you have to read a lot of them to understand what is there they test. The problem of quantity
 - test names very explicit: well, even this sometimes becomes a mess, especially with integration or functional tests, as the names become longer and longer (TestMaskMoveRotateZoomWithUndo() or TestAddMoveRemovePointOnMaskAndCheckDimentionsOnPanelWithUndo() )
 - comments: just like any other piece of code, tests need comments, and they seem to be the best solution to understand what they do. But just like code, people either forget about them, write something just to be there, or forget to update the comments when they update the tests or copy paste.
 - lack of clear flow in tests: preparation, action and verification phases: any good test has 3 steps. One where the context is prepared, then when something is done, and the third is the verification phase. However, there are tests where these 3 aren't easy to spot, and with integration/functional tests especially, because they tend to take time, there are many cases where some of the steps are repeated (add/edit/delete tests for instance, where there is a preparation phase, then action add, verify, action edit, verify, action delete verify). This makes the tests harder to read.

So what do I do?

It seems that BDD tends to solve all the above problems, changing the naming and making test first easier to understand, and my problem by making tests easier to be understood later. It is not perfect, but definately helps. 

But BDD is not just this, BDD is a lot more. BDD for the first time, allows or simplifies the real implementation of the acceptance tests. If you use XP, and you have stories, and for each you define acceptance tests with the customer (see sample http://danbunea.blogspot.com/2008/04/chapter-3-communicating-and.html), which means that both you and him understand them, because they use english and not c, java, ruby or c#, BDD allows you to write them in code very close to the way they are written on the story card. Now this is a breakthough:

Story: import images

The customer will be able to import and process images using the API. 

Acceptance test X:

Given an image over 20Mb
When the user imports it
Then the program should reject it for being too big

would be in code:

   [Test]
  public void ShouldntImportOver20Mb()
  {
        Image image = null;
        ImportResult result = null;
       "Given an image over 20Mb".Given(
         () =>
         {
                image = new Image("largeImage.jpg");
         });


     "When the user imports it".When(
        () =>
        {
             result = Importer.Import(image);
         });


     "Then the program should reject it for being too big".Then(
       () =>
       {
           Assert.Equal("Image is too big, over 20Mb", result.Message);
       });
  }

Now this isn't any fancy BDD framework, but a simple string Extention methods, proposed by Mihai Lazar (brilliantly simple idea), which despite the simplicity of the test, allows me to read easily what is going on, see clearly the flow and see the real correspondence with the acceptance test in the story. 

Conclusion

The example above is too simple, to allow real advantages over TDD, because as a unit test it would be:

   [Test]
  public void ShouldntImportOver20Mb()
  {
           Image  image = new Image("largeImage.jpg");
            ImportResult result = Importer.Import(image);
           Assert.Equal("Image is too big, over 20Mb", result.Message);
  }

which is fairly simple to understand, but what if it were more complicated:


  [Test]
  public void Test2AddMoveRemovePointOnMaskAndPanelDimentionsWithUndo()
  {
  ViewShape view = DragDropShape();
  ShapeEditorControl ctrl = (ShapeEditorControl)view.FindName("shapeEditor");



  int pointsBefore = ctrl.CurrentOption.Points[0].Count;

  //simulate adding new point near to first point
  Point newPoint = new Point(ctrl.CurrentOption.Points[0][0].X + 1, ctrl.CurrentOption.Points[0][0].Y + 5);
  ctrl.InsertThumb(newPoint);


  Assert.AreEqual(pointsBefore + 1, ctrl.CurrentOption.Points[0].Count);
  Assert.AreEqual(1, ctrl.CurrentOption.UndoActionsCount, "added one undo action");

  //move
  Point pointBefore = new Point(ctrl.CurrentOption.Points[0][0].X, ctrl.CurrentOption.Points[0][0].Y);
  Thumb thumb = ctrl.Thumbs[0];
  thumb.RaiseEvent(new DragDeltaEventArgs(3, 3));
  thumb.RaiseEvent(new DragCompletedEventArgs(1, 1, false));

  Assert.AreEqual(pointBefore.X + 3, ctrl.CurrentOption.Points[0][0].X);
  Assert.AreEqual(pointBefore.Y + 3, ctrl.CurrentOption.Points[0][0].Y);
  Assert.AreEqual(2, ctrl.CurrentOption.UndoActionsCount, "added two undo actions");

  ctrl.IsTestMode = true;
  //delete
  MouseButtonEventArgs mbea = new MouseButtonEventArgs(Mouse.PrimaryDevice, 0, MouseButton.Left);
  mbea.RoutedEvent = Mouse.MouseDownEvent;
  ctrl.thumb_MouseDoubleClick(thumb, mbea);

  Assert.AreEqual(pointsBefore, ctrl.CurrentOption.Points[0].Count);
  Assert.AreEqual(3, ctrl.CurrentOption.UndoActionsCount, "added 3 undo actions");


  //Undo
  ClickUndo(view);
  Assert.AreEqual(pointsBefore + 1, ctrl.CurrentOption.Points[0].Count);
  Assert.AreEqual(2, ctrl.CurrentOption.UndoActionsCount, "undone delete");

  //undo
  ClickUndo(view);
  Assert.AreEqual(pointBefore.X, ctrl.CurrentOption.Points[0][0].X);
  Assert.AreEqual(pointBefore.Y, ctrl.CurrentOption.Points[0][0].Y);
  Assert.AreEqual(1, ctrl.CurrentOption.UndoActionsCount, "undone move");

  //undo
  ClickUndo(view);
  Assert.AreEqual(pointsBefore, ctrl.CurrentOption.Points[0].Count);
  Assert.AreEqual(0, ctrl.CurrentOption.UndoActionsCount, "undone add");

  }


and now:

  [Test]
  public void Test2AddMoveRemovePointOnMaskAndPanelDimentionsWithUndo()
  {
     ViewShape view = null;
     ShapeEditorControl ctrl = null;
     int pointsBefore = -1;


     "Given the countour editor open for an image ".Given(
       () =>
       {
                   view = DragDropShape();
                   ctrl = (ShapeEditorControl)view.FindName("shapeEditor");

                 
  int pointsBefore = ctrl.CurrentOption.Points[0].Count;

       });


     "Adding a new point".When(
       () =>
       {
              Point newPoint = new Point(ctrl.CurrentOption.Points[0][0].X + 1, ctrl.CurrentOption.Points[0][0].Y + 5);
              ctrl.InsertThumb(newPoint);

       });

     "should have one more point, and possibility to undo".Then(
       () =>
       {
               Assert.AreEqual(pointsBefore + 1, ctrl.CurrentOption.Points[0].Count);
               Assert.AreEqual(1, ctrl.CurrentOption.UndoActionsCount, "added one undo action");
       });

     Point pointBefore=null;
     "and having a point".When(
       () =>
       {
              pointBefore = new Point(ctrl.CurrentOption.Points[0][0].X, ctrl.CurrentOption.Points[0][0].Y);
       });


     "when moving it".When(
       () =>
       { 
                 Thumb thumb = ctrl.Thumbs[0];
                  thumb.RaiseEvent(new DragDeltaEventArgs(3, 3));
                  thumb.RaiseEvent(new DragCompletedEventArgs(1, 1, false));
       });

     "should be in another position".Then(
       () =>
       {  //move
              Assert.AreEqual(pointBefore.X + 3, ctrl.CurrentOption.Points[0][0].X);
              Assert.AreEqual(pointBefore.Y + 3, ctrl.CurrentOption.Points[0][0].Y);
              Assert.AreEqual(2, ctrl.CurrentOption.UndoActionsCount, "added two undo actions");    
       });

     "when deleting it".When(
       () =>
       { 
              ....
        });
     "should be gone".Then(
       () =>
       { 
              ....
        });

     "clicking undo".When(
       () =>
       { 
              ....
        });
     "should have the deleted point back".Then(
       () =>
       { 
              ....
        });

     "re-clicking undo".When(
       () =>
       { 
              ....
        });
     "should move the point back at initial position".Then(
       () =>
       { 
              ....
        });

     "clicking undo".When(
       () =>
       { 
              ....
        });
     "should get rid of the added point".Then(
       () =>
       { 
              ....
        });


  }

which now looks like a scenario, making it simpler to follow and understand how the application  should behave in this case.

It seems, as a final word, that the first and most productive property of BDD, is that the description of the behavior of the application suddently is in english, and could be read (if you just output the strings on the console) even by the customer. A great leap forward, at least for me. 



Labels:

Friday, May 09, 2008

CHAPTER 6: Quality and Testing

6. Quality and Testing

6.1 Internal and external quality of a system

Defining what quality is in software development, means following the two perspectives of the main stakeholders involved in a software project: the customer’s perspective and the programmer’s perspective.

If we look at defining software quality from the customer perspective, we realize that what the real user sees and feels when interacting with the software, defines the quality. Mary and Tom Poppendieck call it perceived integrity, Martin Fowler and Kent Beck call it external quality of a system.

A wonderful example, regarding external quality of a system, is given in “Lean Software Development: An Agile Toolkit” [], written by Mary and Tom Poppendieck, about how everyone needs to contribute to quality and about how different people see quality differently:

Walt Disney designed Disneyland as a giant stage where several hundred actors make it their job to be sure every guest has a wonderful time. One guest's requirements for having a wonderful time are quite different from the next, and the actors are supposed to figure out exactly what each guest thinks a quality experience should be and make sure he or she has it.

Quality at Disneyland

At Disneyland, even the tram drivers are actors. A friend told me the story of a tram driver who noticed a small girl crying on her way back to the Disneyland hotel. He asked her why she was crying and found out that the crowd around

Mickey Mouse was too large, so the girl had not been able to talk to Mickey.

The driver called ahead, and when the tram arrived at the hotel, there was Mickey Mouse, waiting to meet it. The girl was thrilled, and the driver had done his job of making sure she had a quality experience.

—Mary

Looking at a software product, only from the programmer’s point of view, means mainly looking at the maintainability and extensibility sides and not only if the program respects design patterns or object oriented principles. A software program is of high quality, or it is considered to have a good design or architecture, if it can be modified, adapted and extended with ease.

Martin Fowler and Kent Beck describe internal quality:

… internal quality. This reflects the quality of the internals of the system: how well it is designed, how good the internal tests are, and so on. This is a very dangerous lever to play with. If you allow internal quality to drop you'll get a small immediate increase in speed, rapidly followed by a much bigger decrease in speed. As a result you must keep an eagle eye on this lever and make sure it is always up as far as it can go. Nothing kills speed more effectively than poor internal quality.

[Planning Extreme Programming]

Mary and Tom Poppendieck extend the concept of internal quality to conceptual integrity:

Conceptual integrity means that a system's central concepts work together as a smooth, cohesive whole. The components match and work well together; the architecture achieves an effective balance between flexibility, maintainability, efficiency, and responsiveness.

[Mary and Tom Poppendieck, 2003]

6.2 Automated tests

It is not testing alone that leads to high quality but the constant focus on maintaining quality at its highest, all the time. Traditional processes tend to confuse quality with testing, and although a lot of time is allocated for testing, is the testing phase is left to be the last stage in the process.

Agile processes have developed techniques to maintain focus on the high quality all the time, the most important being continuous integration. At the base of these principles stand automated tests, acceptance tests (defined with the customer) and unit tests that are run frequently giving feedback to the programmers about the quality status of the system. Keeping all tests running at 100%, and having a good coverage of automated tests on the system, focuses the team on quality and detects breaks early so they can be fixed fast before they eat out trough the system.

Automated tests, continuous integration, refactoring and customer involvement, combined with the focus on quality that needs to be delivered at the end of each iteration, ensure that the system’s integrity is built in and maintained all the time, from the beginning to the end.

6.3 Test Driven Development

Ron Jeffries expresses the Test Driven Development paradox, very well:

Writing code and tests is faster than writing just the code, if the code has to work.

Now let’s see if this is true:

The mini TDD experiment

To demonstrate the concepts around test first approaches and test driven development, we will make a small experiment. We assume that we are programmers and we need to code a function that divides two positive numbers. For this experiment we will compare the traditional and the TDD approaches.

Approach #1. Code and fix

As programmers, for a simple division we will write the following “pseudo” code:

Function Divide(No1, No2)

Return No1/No2

For this very simple method, let’s assume we needed 5 seconds to write it. Now let’s test if it works. First we try 6 and 2, expecting 3. It works. Let’s try another combination: 1 and 2, expecting 0.5. It works. Now let’s try 8 and 0. An error just occurred. This means we need to modify the program to display a message to the user that the second number cannot be 0:

Function Divide(No1,No2)

If No2 = 0 then display message “Division by 0 cannot be performed”

Else Return No1/No2

Now let’s test our function again. 6 and 2, result 3, good, 1 and 2, 0.5 as expected, 8 and 0 and a message “Division by 0 cannot be performed” occurs as expected. Now, our program works fine.

Assuming that manual testing is slow and for each combination of numbers we need about 10 seconds, this means that a testing session takes 30 seconds. The total time in which we developed the code was: 5 seconds to write the function, 30 seconds to test and see it has problems with division by 0, then about another 5 to correct the function and 30 minutes to test it again and make sure it works: total 5+30+5+30=70 seconds, a minute and 10 seconds.

Approach #2: Test Driven development

In test driven development, there are a series of steps to write a piece of code, starting with and automated test written first and ending up making that test succeed, by writing the code that it tests. Let’s see how it goes:

Function TestNormalDivision()

Expect 3 as a result of Divide(6,2)

The code above compares the value expected and the value returned by our (yet unwritten code) and if they do not match it fails.

One very important step now is to make sure our test really tests something and it does not work every time, no matter what the code under test does. So for this we need to make sure that when it needs to fail, it fails. So we write the following function:

Function Divide(No1, No2)

Return 0

Now we run the test, and it fails saying: expected 3 but the result was 0. So now we modify the function to return pass the test.

Function Divide(No1, No2)

Return 3

Now we run the test again: 1 test succeeded. Excellent. Now let’s see if it works for 1 and 2, so we update the test:

Function TestNormalDivision()

Expect 3 as a result of Divide(6,2)

Expect 0.5 as a result of Divide(1,2)

We run the test. Failure. Oooh, we just realize the mistake we made (code always returns 3) and modify the Divide function:

Function Divide(No1, No2)

Return No1/No2

Running the test, now passes all our expectations. But now we think, what would happen if we used 8 and 0. Let’s add a new test to the test suite (now we have two) and make sure that if there is division by 0, the user is notified:

Function TestDivisionByZero()

Expect message “Division by 0 cannot be performed” displayed as a result of Divide(8,0)

We run the test. It fails. Now we modify our function to make it work:

Function Divide(No1,No2)

If No2 = 0 then display message “Division by 0 cannot be performed”

Else Return No1/No2

Running all our tests, we discover that they all succeed.

How much time did we need to write this code? We needed 5 seconds to write the first test, 5 seconds to make sure it fails, 1 second to run the test (now testing is done by the computer so we assume it should be at least 10 times faster then manual testing), 5 seconds to modify the code to make the test work, 1 second to run the test, another 5 seconds to extend the test to verify the 1,2 combination, 1 second to see that the test fails, 5 seconds to modify the function and 1 second to see it working, another 5 seconds to write the second test and 2 seconds to see the first test work but the second failing, and 5 seconds to complete the code and another two to run the 2 tests and make sure it works. Wow, a long way: 5+5+1+5+1+5+1+5+1+5+2+5+2 = 42 seconds.

Using both approaches, we ended up with the same code. The amount of code written for the second approach is bigger then for the first, having the code and the tests. The amount of time needed for the second approach was arguably smaller then the amount for the first approach, which leads us to Ron Jeffries’s conclusion: to obtain good code, writing tests and code is faster then code alone. The main advantage is that we use computer power to do the testing rather then human power, so we are much faster. Then we can run the automated tests over and over again and it will take 2 seconds to see if they work, manually it will take 30 to do the same thing.

Let’s go further with our experiment, assuming that now we need to extend the program to be able also to do addictions, subtractions and multiplications.

Approach #1. Since all these operation are not affected by 0, but we test that anyway, the code written first will work, so it would take about 5 seconds to write each method, and testing each with 3 combinations of numbers would result in about 30 seconds to test each. The amount of time, needed would be 5+30+5+30+5+30 = 105 seconds, 1 minute and 45 seconds. Testing the whole program (the 3 new methods and the division method) would take us 4*30 = 120 seconds, which is 2 minutes.

Approach #2. TDD

Operations just as above will need only one test, checking 3 combinations. Let’s say it takes 10 seconds to write a test method like this:

Function TestMultiplication()

Expect 0 as a result of Multiply(6,0)

Expect 3 as a result of Multiply(3,1)

Expect -9 as a result of Multiply(3,-3)

Then we’d have to make sure it fails: 5 seconds, 1 second to run the test, then we’d write the code to make it work: 5 seconds and 1 second to make sure it works, so it takes about 10+5+1+5+1 = 22 seconds for each new function, resulting in 3*22 = 66 seconds or 1 minute and 6 seconds to write the new functions. Testing all the code would mean running 5 test methods (2 for division and 1 for the other three), which would be run in 5 seconds.

Tests and code, faster then just code

Comparing the times needed to test our incredibly simple system: 2 minutes vs. 5 seconds show us that not only the code is written faster (110+105=215 seconds vs 66+45=111 seconds), but making sure it works requires far less time for the TDD approach. And second big advantage, it can be done by a computer.

Using a continuous integration machine that downloads the program sources and runs the test suite, then sends us an email telling us what happened, means 5 seconds for the machine and 0 seconds on my side to test the whole system. Using the first approach, would take me 2 minutes to make sure the whole system works. I could delegate this responsibility of testing the whole system to the testing team, but the feedback times, telling me whether the system works as a whole or not, increase rapidly to days and weeks and by that time I should be doing something else.

Scalability

In the 3rd phase of our little experiment, we analyze what would happen if our system would have 400 functions instead of 4. Using the first approach it would need about 12000 seconds (that is over 3 hours) for a full test, while using the TDD and automated testing suite about 500 seconds or, better said, less then 10 minutes. This simple sample shows us scalability when it comes to TDD vs traditional coding approaches. The testing team could work, to some extent in parallel, but after, all I could set my integration machine to divide the tests and work in parallel.

After making this very small experiment, we showed how test driven development is, compared with just coding:

- faster to develop

- faster to test the whole system and give feedback

- scalable

Tests as documentation

Another advantage of the method described above is that, the automated tests can act as a very good documentation of the code written. In traditional approaches, just documenting things that can be very easily deduced from the automated tests, like how a function works, would increase even more the development time. After all, just reading:

Function TestDivisionByZero()

Expect message “Division by 0 cannot be performed” displayed as a result of Divide(8,0)

tells me or someone new in the project, that if you try to perform a division by 0, the system will display an error message on the screen.

Embrace change: how?

Having a system with 4, 400, 1000…100.000 methods, doesn’t really comfort me when it comes to making a change in it. If I change one tiny piece of code somewhere, could I break something in another part of the system? And if I do, how could I know fast enough, to be able to either correct it or reverse my changes?

To have the feedback from the code, telling me if and where I’ve broken some existing functionality, I would normally need to retest the whole system. For a 4 methods system, it would take 2 minutes, but for a more likely system, it might take hours, days or even weeks. So the courage to change decreases with the system getting bigger, thus is shortening the life of the system. When a system is too rigid and can no longer adapt to the changes on the market, it is bound to die.

Having a full regression automated test suite, that runs very fast and can be run very often, means fast feedback. Fast feedback means changes are less risky and can be done easier and faster, thus extending the life of a system.

Design advantages

Another advantage of writing automated tests for the code is that the code written tends to be very loosely coupled, thus better designed. Test driven development also tends to eliminate “partially completed code”, encouraging less code to be written, as the programmer is more focused on what is really needed, thus decreasing the amount of code and its complexity.

At a macro level, the fact that changes, even in the architecture are much easier to be performed, when using TDD combined with aggressive refactoring, allows the programmers to continuously upgrade the design and update the architecture. Since the changes are easy to do, the evolutionary design technique is encouraged, having a much smaller need to build a flexible architecture upfront, following the YAGNI principle from XP.

How do we achieve external quality at 100% all the time?

After each iteration, a potentially shippable system is delivered, so quality is at 100%, every 2-4 weeks, maintaining this way the system at the highest level throughout its lifecycle.

Frequent releases that allow the customer representatives to see the system all the time are maintaining the project on the right path and at the right perceived quality. When the customer sees the software he can very easily spot things that are not according with what he thought they would be, and together in the next iteration these issues can be fixed. The focus on quality in agile development is mainly expressed by the iterative and incremental development model, which allows the stakeholders in a project to identify problems early and correct them as soon as they are found, thus maintaining the system on the highest quality.

As a small sample, we think about one frequent non-functional requirement: the system must be fast. In traditional testing techniques, the test and development teams had to figure out by themselves what fast means. This is a very difficult requirement to deal with, because it is almost impossible to measure. Presenting the system, after an iteration, to the customer, might let him say: “I want that report to be faster. At this time it needs about 10 seconds. Can you make it faster? Let’s say 2-4 seconds”. Suddenly, there is a clear definition on where “fast” applies, allowing the team to focus on solving a concrete problem. This is a simple sample, showing how perceived quality is maintained by the continuous collaboration and feedback between the customer and the team.

Who is involved in quality, how and when?

Everyone, all the time. In traditional systems, the responsibility for quality is mainly delegated to testing teams that must make sure the code is of high quality. Agile thinking makes quality a collective responsibility of the customer, the developers and the testers all the time from the first, to the last minute of a project.

The customer is involved in quality by defining acceptance tests. The developers are involved in quality by helping the customers write the tests, by writing unit tests for all the production code they write and the testers are involved by helping the developers automate acceptance (customer) tests and by extending the suite of automated tests.

Manual vs. automated testing

Manual testing is not forgotten, although in agile methodologies a great emphasis is put on automated tests. Manual testing is still performed. Agile testing is about balancing between the need to automate when it is beneficial and relying on manual testing when it is more efficient then writing automated tests.

How do we achieve internal quality?

Simplicity, unit tests running at 100%, refactoring, YAGNI, collective ownership, continuous integration combined with good coding techniques, are all practices that as a whole are aimed at maintaining the code quality at its highest all the time.

Regarding code quality and the simplicity principle, Ron Jeffries writes:

Everything we write must:

  1. Run all the tests
  2. Express every idea we need to express
  3. Say everything once and only once
  4. Have a minimum number of classes and methods, consistent with the above

[Ron Jeffries, 2001]