A few days ago, Microsoft Research released a news article, “Exploding Software-Engineering Myths“, debunking several software engineering myths and discovered/confirmed a few beliefs using actual empirical data.
The biggest problem for several researchers and software engineers were a lot of their hypothesis and conjectures are based on anecdotal and personal experience. These were seen to be true throughout several projects, but none were able to confirm or deny these without empirical data… Until now. This research was done by Nachi Nagappan, a senior researcher at Microsoft Research Redmond with the Empirical Software Engineering and Measurement Research Group (ESM). Working with Microsoft and their various (enormously huge number of) development teams, Nagappan was able to confirm several beliefs and even debunk and correct some of these hypothesis.
Some of the discoveries are shocking. Here’s a brief summary of the article:
Code coverage != software quality
You should spend more time testing complex code and more used code rather than having a high code coverage.
TDD does takes more time to release, but reduces post release maintenance cost
TDD actually produces better quality code, but takes 15% to 35% longer to complete. However, it reduces the cost after release significantly. (Realizing quality improvement through test driven development: results and experiences of four industrial teams)
More assertions and code verifications means fewer bugs
The density of assertions/contracts actually have a negative correlation to code defects. However, forcing the use of assertions will not work; building a culture will. This is a good thing because Code Contracts will be released in .NET Framework 4.0. (Assessing the Relationship between Software Assertions and Code Quality: An Empirical Investigation)
Organizational metrics can predict software failure-proneness
The software system developed will usually resemble the organization building the system (Conway’s Law). This is really startling for me to find out various factors that organizational faces will actually affect the software’s quality and failure rate. It means that there is a direct correlation between organizational structure and the quality of software. The Influence of Organizational Structure on Software Quality: An Empirical Case Study
Distance does not matter
Most believe that distributed-development model has negative impact to software quality. But surprisingly enough, the impact is statistical negligible. In fact, organizational cohesiveness plays a more important role than distance or geographical location. (Does distributed development affect software quality? An empirical case study of Windows Vista)