Code generation friend or foe?
Usually in the agile world code generation is considered wrong. In order to be able to contradict this belief we must go to its roots.
In my opinion, code generation has two main reasons why is usually considered wrong:
1: Tools that generate uncontrollable code
Throughout the years, many IDEs and tools generated code. For the beginners and for marketing people this was the panacea but experienced developers quickly came to the conclusion, that these tools were doing more damage then good. This was usually because the generated code either underperformed, or was poorly written, generation could not be properly controlled or as soon as the application was getting more complex, the generated code was far from a solution. Debugging and changing generated code was difficult. This led to the raise and fall of CASE tools.
The only code generation tools that seem to have lasted over the years are the UI designers, although there are still disputes whether you should do the HTML 'by hand' or using a tool like Dreamweaver or FrontPage. My opinion is that probably as in most cases, the truth is somewhere between. Use the tool, but also control everything by hand.
2. People generate code before they can afford it
Sometimes, beginners start generating the code before really understanding what it must do. For instance, recently I saw a few fellow developers adopting a new ORM and found out that there is a tool that can generate the mapping xml files. They started with the tool, but as they didn't really have the knowledge to understand what should be generated, how this mapping files really work, ended up in redoing the mapping files by hand and taking everything step by step just to understand how it works.
The good
In my opinion code generation should be applied in a totally different
way, and this way is in my opinion the agile way.
Steps to use code generation as an agile technique:
STEP 1. Develop by hand and get a very good understanding of the domain and
of the code
STEP 2. If the code can be generated, then why not.
Code generation, can drastically improve the speed of a project. This is a very powerfull weapon that developers can use, but is preferable to be carefully studied before starting to shoot it. Just like any powerful weapon, if used wrong it can have devastating effects.
It is very useful in changing environments, when parts
or the code when a change is done is generated. This is especially
useful in testing user interfaces. Much code can just be regenerated,
when the interface changes, which makes maintainability much easier, and thus automating user interface tests much more friendly.
Conclusion
Because code generation, can help in changing environments, can help the speed of development I think that code generation is the big missing agile technique from any methodology. But like most agile techniques that are also very powerful should be approached with care , remembering that it does not replace coding manually , just help it and it can by no means be used in environments where the base knowledge has not been acquired yet.
List of available code generators for .NET:
http://sharptoolbox.com/Categoryb4d647ba-2494-485a-b75b-8f8f3046d3ab.aspx
Thursday, May 19, 2005
Thursday, May 05, 2005
Self training
In a world, where because of the globalization, there are people that don’t get a very good night sleep, the quest for efficiency is more acute then ever. In the world of software development, efficiency can be gained by two primary factors:
-reuse
-improve your developers
If for the primary, the best answer lies in front of you and comes from Microsoft, (just think about why IE couldn’t be removed for good from Windows), for the second there is still no clear and single answer.
Short “knowledge spreading” meetings
One way to improve the knowledge level, that proved very useful to us, is to have short meetings where different things, from pure technical to methodology practices, are discussed in front of a whiteboard. Agile teams, where everyone has something to offer, make this technique very beneficial, as combined with osmotic communication (See Alistair Cocburn's Crystal Clear), it is a great way to spread knowledge. When doing this kind of meetings, from the experience we have, there are a few things that need to be taken into consideration:
- meetings must have a plan of what will be discussed
- meetings must be short
- everyone can propose subjects to be discussed
- everyone has to be involved
- have a whiteboard
- make sure you really have something to talk about (ideally probably you should have one meeting every week, not less)
In any agile team, open communication must be encouraged at the maximum. This is one very good technique of doing it.
School like sessions
One problem I noticed after adopting a new technology (NHibernate – http://nhibernate.sf.net – some of us were familiar with it or other ORMs), and using it in several small web projects, was that although it was used for quite a while (two months) it was misused, as not too many things about its possibilities were known. I also felt as if with each new project and team adopting this new technology, the knowledge was weaker and all misusages from the early projects were propagated to the others. We even got to a point where a colleague told us that he doesn’t really like using it and we saw that he was reusing some badly written pieces of database layer code, from earlier projects. I saw then that the problems were not with the technology but with lack of knowledge about it.
The reason for this might be time pressure (I don’t really buy this one) or , a more plausible one, that the developers of the second project took the code from the developers that developed the first one, as “good”, as a reference of how thing with NH should be done, trusting the first team and reusing the “misused” code.
Then I decided that it is time for a new meeting looking trough the code one day, but then it stuck me: how about making a school like activity, where everyone reads about a thing, then have a small test to see what the understood from it, and if the level of understanding isn’t satisfactory re-explain everything. Said and done. I decided that at first I should make a small introduction about the most basic things in the technology for the people that haven’t used it yet, then give everyone an article/book chapter that seems very relevant when you want to explain a new technology, going deep into it, then after the reading session is over, everyone gets questionnaire, and sends back the answer to see if they understood correctly what was needed, and if they really found the hot spots in the material. After analyzing the answers, from everyone, all the answers were discussed with everyone in front of the whiteboard, to make sure that everyone understands what needs to be understood.
Is it worth it?
The results were very pleasing. For two reasons: everyone after only a short time had a good understanding of the new technology (transient/persistent/detached states, moving an object from one to another, hql, criteria api, eager/lazy fetching, associations etc) and since it is not a requirement to memorize everything , at least, when confronting a problem, the developers will know where to look for, they will have a good image of the documentation, and where to look for specific answers. I strongly believe that even if that this meeting can take a couple of hours, this time is very quickly recovered by having people know what they do and have less bugs and problems the kind we has with misusage.
Another very interesting fact, was that after two such sessions, everyone felt very happy, knowing something new, or some intimate details, making them more motivated and more keen to start working with it. It made developers happy. Remember my article about sharing the thrills?
The knowledge wiki
At the suggestion of a dear friend (which can be found in the comments), we installed a wiki. Actually it is not a wiki, but an simple CMS system, developed internally, where people can share information, and where information has something that is lacked in the first two approaches: persistence. A wiki, has the main purpose in my opinion to keep the links where people can read more, a common place for links, a common 'favourites' system and also, and most importantly, a place where the hard to find information can be shared and persisted, information like, bugs and false assumptions about new technologies, samples for different internally developed technologies or other deep things.
Whether this will be a sucess or not we will definately see, but although the system has been installed, just for a few hours, it seems to be a success, as everyone seems very interested in both reading and writing content on our wiki, and even suggestions about extending it with a forum appeared.
Some screenshots can be found at:
Wiki and Administration
I certainly hope that besides this techniques, there are others that wait to be discovered in our big goal of become better, by self training and better knowledge sharing, and that this discussion does not end here ...
-reuse
-improve your developers
If for the primary, the best answer lies in front of you and comes from Microsoft, (just think about why IE couldn’t be removed for good from Windows), for the second there is still no clear and single answer.
Short “knowledge spreading” meetings
One way to improve the knowledge level, that proved very useful to us, is to have short meetings where different things, from pure technical to methodology practices, are discussed in front of a whiteboard. Agile teams, where everyone has something to offer, make this technique very beneficial, as combined with osmotic communication (See Alistair Cocburn's Crystal Clear), it is a great way to spread knowledge. When doing this kind of meetings, from the experience we have, there are a few things that need to be taken into consideration:
- meetings must have a plan of what will be discussed
- meetings must be short
- everyone can propose subjects to be discussed
- everyone has to be involved
- have a whiteboard
- make sure you really have something to talk about (ideally probably you should have one meeting every week, not less)
In any agile team, open communication must be encouraged at the maximum. This is one very good technique of doing it.
School like sessions
One problem I noticed after adopting a new technology (NHibernate – http://nhibernate.sf.net – some of us were familiar with it or other ORMs), and using it in several small web projects, was that although it was used for quite a while (two months) it was misused, as not too many things about its possibilities were known. I also felt as if with each new project and team adopting this new technology, the knowledge was weaker and all misusages from the early projects were propagated to the others. We even got to a point where a colleague told us that he doesn’t really like using it and we saw that he was reusing some badly written pieces of database layer code, from earlier projects. I saw then that the problems were not with the technology but with lack of knowledge about it.
The reason for this might be time pressure (I don’t really buy this one) or , a more plausible one, that the developers of the second project took the code from the developers that developed the first one, as “good”, as a reference of how thing with NH should be done, trusting the first team and reusing the “misused” code.
Then I decided that it is time for a new meeting looking trough the code one day, but then it stuck me: how about making a school like activity, where everyone reads about a thing, then have a small test to see what the understood from it, and if the level of understanding isn’t satisfactory re-explain everything. Said and done. I decided that at first I should make a small introduction about the most basic things in the technology for the people that haven’t used it yet, then give everyone an article/book chapter that seems very relevant when you want to explain a new technology, going deep into it, then after the reading session is over, everyone gets questionnaire, and sends back the answer to see if they understood correctly what was needed, and if they really found the hot spots in the material. After analyzing the answers, from everyone, all the answers were discussed with everyone in front of the whiteboard, to make sure that everyone understands what needs to be understood.
Is it worth it?
The results were very pleasing. For two reasons: everyone after only a short time had a good understanding of the new technology (transient/persistent/detached states, moving an object from one to another, hql, criteria api, eager/lazy fetching, associations etc) and since it is not a requirement to memorize everything , at least, when confronting a problem, the developers will know where to look for, they will have a good image of the documentation, and where to look for specific answers. I strongly believe that even if that this meeting can take a couple of hours, this time is very quickly recovered by having people know what they do and have less bugs and problems the kind we has with misusage.
Another very interesting fact, was that after two such sessions, everyone felt very happy, knowing something new, or some intimate details, making them more motivated and more keen to start working with it. It made developers happy. Remember my article about sharing the thrills?
The knowledge wiki
At the suggestion of a dear friend (which can be found in the comments), we installed a wiki. Actually it is not a wiki, but an simple CMS system, developed internally, where people can share information, and where information has something that is lacked in the first two approaches: persistence. A wiki, has the main purpose in my opinion to keep the links where people can read more, a common place for links, a common 'favourites' system and also, and most importantly, a place where the hard to find information can be shared and persisted, information like, bugs and false assumptions about new technologies, samples for different internally developed technologies or other deep things.
Whether this will be a sucess or not we will definately see, but although the system has been installed, just for a few hours, it seems to be a success, as everyone seems very interested in both reading and writing content on our wiki, and even suggestions about extending it with a forum appeared.
Some screenshots can be found at:
Wiki and Administration
I certainly hope that besides this techniques, there are others that wait to be discovered in our big goal of become better, by self training and better knowledge sharing, and that this discussion does not end here ...
Tuesday, May 03, 2005
Agile databases - embrace change
Ever changing databases
In a world where we have to adopt new processes to develop software in order to adapt to ever changing environements, what happens to our databases. Do our databases, keep up the rhytm with the code in changing conditions?
Although we made big progresses in 'becoming agile' we are still at the beginning although instant results were seen ever since we started. But although there are lots of materials about agile processes, about agile techniques that are focused on making agile processes work like unit testing and TDD, refactoring, continuous integration, frequent delivery, evolutionaty design etc there are few materials over the web that deal with databases in agile processes.
I would like to say that mostly we are confronting ourselves with simple databases (<=50 tables).
The questions I have are:
1. How do databases deal with changes?
2. How to develop tests that are easely maintenable in these conditions?
1. How do databases deal with changes?
After adopting a two week iteration technique, we were confrunted with many versions of a database, one every two weeks. The database was evolving and often changed. The problem is bigger if the system has gone live. You must make an update to the live database with every new version. In the first iterations of a project, usually until a 'stable' database schema is adopted, it changes a lot, but if the system doesn't go live this is a minor problem as old databases can be just dropped and rewritten. After the stable version is reached, the updates from one version to another can be done more easely with alter scripts.
From time to time the client told us they were having difficulties with the new version we just sent, as it just crashes immediately after start. We noticed that this is usually because the database scripts haven't been run and the program is the new version, but the database is old and it the application just crashes at some time. This has given us a lot of sweats and afterall,clients that came to see the new version and see it crashing remember that your software is buggy not that this has something nothing to do with you.
Versions table and update scripts
A way to deal with this was the versions table and update scripts. Each modification in the database from one version to another version is made with a script and not directly. The database also contains a Versions table the logs all these updates. This way we can know what is the current database version, when was it created , what modifications were made (a comments field) who did the script and when it was actually run. How do we know this? By simply putting an insert line into our update script that puts in this data into the versions table.
After applying this techinque we thought about putting in the application installer, a part of code that runs the update. And we did it, but new problems arrived. A client managed somehow to override a live database, although we put in a lot of messageboxes asking to confirm whatever he did. So we thought of adding a line in the configuration file of the application which says which version should the database have in order to start the program and unless the database has the right version, the program won't start. Now noone forgets to make the updates.
Automating the update script creation
This is a simple solution that seems to work well. However a problem with it is that manually creating the update scripts is taking a lot of time to do and to test. This problem brought to our attention a product, build by RedGate, that knows to generate the update script after comparing two databases. This greatly reduces the amount of work, needed to make the update scripts, we can now make them manually on a test database then with the tool generate the update script.
2. How to develop tests that are easely maintenable in these conditions?
Everyone teaches us to make our database related tests, data independent. The most used techniques are made either by having a separate database used for testing , creating the test database with data on setup and dropping it on teardown and using a rollback technique (database server rollback or using Entreprise Services). Each of these techniques have their own problems, although seem to solve the problem of making the database tests independent from the data in the database. But are they easely maintenable if the database stucture changes often?
ORMs
It seems that using ORMs makes database access code more testable and easely maintenable, as errors after a change was made is detected fast. Some ORMs even know how to generate the create, and most importantly the update scripts themselves. If you have the unit tests related to the ORM's persistent objects the errors in tests can be detected very fast, first just by compiling then by running them so that query problems are revealed. ORMs seem in my opinion more agile then other approaches as using ADO.NET connected and disconnected database access. Is this the solution for all problems? No, however they seem to embrace change more easely.
Regression query integrity tests
Whether you are using ORMs or not, when making a structure change to a database you need instant feedback to see if you broke anything. It is very important to know that at least,that all sql queries (queries/stored procedures etc) work. Not that they work ok, but that they work. It is very important to know this immediately after making the changes. So maybe one technique is to make a test suite that just verifies validity of sql queries and not functionality. I call these regression query integrity tests.
My idea is to create a list of very simple tests, that just run the queries, to see if they break or not. This can be easily run each time, a structural change in the database is done, to see if something was broken or not. Some of the code for this kind of tests can be generated. Code generation, is in my opinion the great missing piece from any agile metholodology's list of practices, as by using code generation, productivity can dramatically improve.
Conclusion
Although we develop projects that are quite simple, from a DBAs point of view, we must keep in mind that it is very important to have techniques that enable us to change the database code, just as easy as the code itself, otherwise this can rapidly become the big bottleneck of our project, that 'embrace chage'.
In a world where we have to adopt new processes to develop software in order to adapt to ever changing environements, what happens to our databases. Do our databases, keep up the rhytm with the code in changing conditions?
Although we made big progresses in 'becoming agile' we are still at the beginning although instant results were seen ever since we started. But although there are lots of materials about agile processes, about agile techniques that are focused on making agile processes work like unit testing and TDD, refactoring, continuous integration, frequent delivery, evolutionaty design etc there are few materials over the web that deal with databases in agile processes.
I would like to say that mostly we are confronting ourselves with simple databases (<=50 tables).
The questions I have are:
1. How do databases deal with changes?
2. How to develop tests that are easely maintenable in these conditions?
1. How do databases deal with changes?
After adopting a two week iteration technique, we were confrunted with many versions of a database, one every two weeks. The database was evolving and often changed. The problem is bigger if the system has gone live. You must make an update to the live database with every new version. In the first iterations of a project, usually until a 'stable' database schema is adopted, it changes a lot, but if the system doesn't go live this is a minor problem as old databases can be just dropped and rewritten. After the stable version is reached, the updates from one version to another can be done more easely with alter scripts.
From time to time the client told us they were having difficulties with the new version we just sent, as it just crashes immediately after start. We noticed that this is usually because the database scripts haven't been run and the program is the new version, but the database is old and it the application just crashes at some time. This has given us a lot of sweats and afterall,clients that came to see the new version and see it crashing remember that your software is buggy not that this has something nothing to do with you.
Versions table and update scripts
A way to deal with this was the versions table and update scripts. Each modification in the database from one version to another version is made with a script and not directly. The database also contains a Versions table the logs all these updates. This way we can know what is the current database version, when was it created , what modifications were made (a comments field) who did the script and when it was actually run. How do we know this? By simply putting an insert line into our update script that puts in this data into the versions table.
After applying this techinque we thought about putting in the application installer, a part of code that runs the update. And we did it, but new problems arrived. A client managed somehow to override a live database, although we put in a lot of messageboxes asking to confirm whatever he did. So we thought of adding a line in the configuration file of the application which says which version should the database have in order to start the program and unless the database has the right version, the program won't start. Now noone forgets to make the updates.
Automating the update script creation
This is a simple solution that seems to work well. However a problem with it is that manually creating the update scripts is taking a lot of time to do and to test. This problem brought to our attention a product, build by RedGate, that knows to generate the update script after comparing two databases. This greatly reduces the amount of work, needed to make the update scripts, we can now make them manually on a test database then with the tool generate the update script.
2. How to develop tests that are easely maintenable in these conditions?
Everyone teaches us to make our database related tests, data independent. The most used techniques are made either by having a separate database used for testing , creating the test database with data on setup and dropping it on teardown and using a rollback technique (database server rollback or using Entreprise Services). Each of these techniques have their own problems, although seem to solve the problem of making the database tests independent from the data in the database. But are they easely maintenable if the database stucture changes often?
ORMs
It seems that using ORMs makes database access code more testable and easely maintenable, as errors after a change was made is detected fast. Some ORMs even know how to generate the create, and most importantly the update scripts themselves. If you have the unit tests related to the ORM's persistent objects the errors in tests can be detected very fast, first just by compiling then by running them so that query problems are revealed. ORMs seem in my opinion more agile then other approaches as using ADO.NET connected and disconnected database access. Is this the solution for all problems? No, however they seem to embrace change more easely.
Regression query integrity tests
Whether you are using ORMs or not, when making a structure change to a database you need instant feedback to see if you broke anything. It is very important to know that at least,that all sql queries (queries/stored procedures etc) work. Not that they work ok, but that they work. It is very important to know this immediately after making the changes. So maybe one technique is to make a test suite that just verifies validity of sql queries and not functionality. I call these regression query integrity tests.
My idea is to create a list of very simple tests, that just run the queries, to see if they break or not. This can be easily run each time, a structural change in the database is done, to see if something was broken or not. Some of the code for this kind of tests can be generated. Code generation, is in my opinion the great missing piece from any agile metholodology's list of practices, as by using code generation, productivity can dramatically improve.
Conclusion
Although we develop projects that are quite simple, from a DBAs point of view, we must keep in mind that it is very important to have techniques that enable us to change the database code, just as easy as the code itself, otherwise this can rapidly become the big bottleneck of our project, that 'embrace chage'.
Subscribe to:
Posts (Atom)