Archive
Code Coverage with PartCover
I’ve been getting more into Test Driven Development (TDD) recently, so I’ve been building up my unit testing toolset as I go along. I’ve got the basics covered – the industry-standard NUnit as my testing framework & test runner, and Castle Windsor if I need a bit of IOC, are the main things.
But I’m at the stage now where I know the basics so I’m trying to improve my code and my tests, as because anyone in TDD will tell you, if your tests are no good, they may as well not be there. So the next obvious step was a code coverage tool. At the basic end, this is a tool that you use to find out how much of your application code actually gets tested by your unit tests, which can be a useful indicator of where you need more tests – although it doesn’t say whether your tests are any good or not, even if you have 100% coverage.
Open Source Tools
Being Scottish by blood, I’m a bit of a skinflint, so like most of my other tools I wanted something open source, or at least free. This actually proved more difficult than I first imagined – most of the code coverage tools seem to be commercial, often part of a larger package. Some also required that you alter your code to use them, by adding attributes or method calls – definitely not something I wanted to do. The main suitable one I came across was PartCover, an open source project hosted on Source Forge. Thankfully, it was easy to install and set up, and did everything I wanted.
Using PartCover
There are two ways of running PartCover – a console application, which you would use with your build process to generate coverage information as XML, and a graphical browser, which lets you run the tool as you like and browse the results. I should point out I haven’t tried the console part, or using the generated XML reports, which you would probably want to do in a larger-scale development environment along with Continuous Integration etc.
Either way, the first thing you have to do is configure the tool – you need to define what executable it will run, any arguments, and any rules you want to define about what assemblies & classes to check (or not):
PartCover Settings
You can see here that I’m running NUnit console – you don’t actually have to use this with unit testing. If you want, you can just get it to fire up your application, use your app manually, and PartCover will tell you what code was run – which I imagine could come in quite handy in itself in certain situations. But as I want to analyse my unit test coverage, I get it to run NUnit, and pass in my previously created NUnit project as an argument.
Rules
I also have some rules set up. You can decide to include and exclude any combination of assemblies and classes using a certain format with wildcards – here I’ve included anything in the ‘ContactsClient’ and ‘ContactsDomain’ assemblies, apart from any classes ending in ‘Tests’ and the ‘ContactsClient.Properties’ class. It can be useful to exclude things like the UI if you’re not testing that, or maybe some generated code that you can’t test – although you shouldn’t use this to sweep things under the carpet that you just don’t want to face!
Analysis
With that done, just click ‘Start’ and you’re away – NUnit console should spring into life, run all of your tests, and you’ll be presented with the results:
PartCover Browser
As you can see, you get a tree-view style display containing the results of all your assemblies, classes and methods, colour coded as a warning. But that’s not all! Select ‘View Coverage details’ as I’ve done here, and you can actually see the lines of code which have been run, and those which have been missed. In my example above, I’ve tested the normal path through the switch statement, but neglected the error conditions – it’s exactly this type of thing that code coverage tools help you to identify, thus enabling you to improve your tests and your code.
At this point, I feel I have to point out a potential issue:
Warning! Trying to get 100% test coverage can be addictive!
If you’re anything like me, you may well find yourself spending many an hour trying to chase that elusive last few percent. This may or may not be a problem depending on your point of view – some people are in favour of aiming for 100% coverage, but some think it’s a waste of time. I like the point of view from Roy Osherove’s Book, that you should test anything with logic in it – which doesn’t include simple getters/setters, etc.
But 100’s such a nice, round number, and I only need a couple more tests to get there..
The Mythical Man-Month
As part of my learning drive, I decided to read some of the books I’ve been hearing about. The first one I decided to tackle was the often-quoted Mythical Man-Month (TMMM) by Frederick Brooks, originally written in 1975 based on Brooks’ experience managing development of systems including IBM’s Operating System/360. This was a time when mainframes roamed the earth, assembler was still the language of choice for many, and memory cost around $100,000 a Meg. Although I thought the book would be somewhat outdated, with it’s small size (at 300-odd pages) I figured I might actually be able to finish it this side of Christmas!
The Mythical Man-Month Cover
Well, some of it is indeed out of date – there is much talk of assembler, 5-foot tall system manuals, batch programming, lack of system space/time, and other problems associated with the huge expense of computer systems back then. Some is also applicable only to really big systems – at it’s peak, the OS/360 team employed over 1000 people, and over 5000 man-years went into it’s development & maintenance! And I thought our ACME work management system was big..
But the surprising thing is how relevant most of the book still is today – indeed, there are still active online discussions regarding many of the points. I would say of the original 15 chapters, at least 10 of them are still totally applicable today, as they concentrate on the human side of software management. More than that, they contain lessons that many of the IT managers and developers I’ve worked with would do well to learn. In fact, in some ways it’s quite disheartening to realise that the things we still struggle with today are the same as those we struggled with back then – in many ways, it seems we’re no better off now than we were more than 30 years ago. It’s not so much that there are no solutions to the problems, more that there are no easy solutions – software development is hard. And it’s also an essentially different beast to the kind of things we’ve had to manage before, such as building projects, and yet many people still approach it using the same techniques and expect the same results. Of course, it’s still a young industry, so perhaps if a few more people read this book and those like it, in another 30 years, we might be starting to get somewhere..
Anyway, below I’ve discussed a few of the more pertinent points raised in the book. I must warn you, this is a rather lengthy post – but then it is a book full of good ideas. I mean, how many books can you name that are still so well known nearly 35 years after they were published, or ones that have a law named after them for that matter?! So I hope you have a comfy chair.
Estimation
As most developers would agree, software is inherently difficult to estimate – but it’s not always clear why. Early on in the book, Brooks gives some of the reasons behind this – as with points throughout the text, backed up by impartial research where possible:
- Developers are Optimists – we tend to imagine things going well. While this may happen for an individual piece of work, the chances of everything going well on a project are negligible, but instead of taking this into account we tend to add up all of the ‘best-case scenario’ estimates and arrive at an unrealistic overall figure.
- Developers Consider Isolated Development – on seeing some new application or website, you’ll often hear a dev say ‘that’s easy, I could code that in a weekend!’. I’ve heard this many times, even from experienced developers. The fact is, knocking up a quick program that works on a single machine is generally easy – it’s turning this into a polished, tested, documented product that works in a full-scale environment that takes the time.
- Men and Months are not Interchangeable – certain tasks can be split up amongst people and retain the same efficiency. The sort of tasks this applies to are those with no communication required, no overlap between the different partitions – sewing a field, for example. Writing software is not such a task. As more people get involved, more communication is required, and efficiency drops – in fact at a certain point, adding more people can start to increase the time taken to complete the overall task. This is often not taken into account, and projects are planned with a certain number of ‘man-months’ divided between an arbitrary number of workers, with the end result being missed deadlines.
- Productivity doesn’t increase Linearly – as studies into large-scale developments have shown, the rate of work decreases as systems grow. Data show an exponent of around 1.5, so as system complexity increases (as measured in lines of code), it takes increasingly more effort to add extra functionality. This isn’t something most people take into account, but can make a huge difference for larger developments.
- Not all Time is Used – in further studies, when time was recorded at a low level, it was found that only about half of a developer’s time was spent on project development, with the other half occupied by meetings, unrelated work, personal items, and so on.
While most people are aware of the general problems, if not the specific causes, I for one always seem to end up on projects with unreasonable deadlines. I think this is a combination of the factors above, along with what Brooks alludes to in the book – that it can be very difficult to explain to a customer why it’s going to take such a seemingly long time to create what appears to be a simple application. Until we have a more solid foundation for software estimation (which doesn’t seem forthcoming), software managers need to trust their experience and give realistic dates, and not bow to client pressure and give ridiculously optimistic dates that aren’t going to help anybody in the long run.
Project Overrun
Related to the difficulties in estimation is the often-cited problem of software projects often going way, way, waaaaaay past their initial due date – usually with enormous increases in budget to boot. As well as estimation issues, Brooks gives several other factors which contribute to this problem – although he later points out that most of these only really apply to waterfall style development, which has been superseded by incremental development in many cases (which someone would tell my manager..)
One such factor, which is extremely common if my experiences are anything to go by, is that the deadlines are often set by the customer. This is likened to the cooking of an omelet – the customer might say he wants his omelet in two minutes. And after two minutes, if the omelet is not cooked, he can either wait – or eat it raw. The cook may try to help by turning up the heat, but you’ll just end up with an omelet which is burned on the outside and raw on the inside. Sadly, I’ve worked on a couple of projects where the customer ended up with a burned/raw application because ‘it has to be finished by X!’
The book also suggests that most projects don’t give enough time to testing, which can be a fatal mistake. A lot of issues are not found until the testing phase, by which time the deadline has nearly arrived – slippage here comes as a surprise to everyone, as things seemed to be going well, and once the wheels of enterprise have been put into motion, late delivery can have serious financial implications. Brooks even gives his breakdown of project times (again, this only applies to waterfall) – 1/3 planning, 1/6 coding, 1/4 component test, and 1/4 system test.
Also discussed is what happens when it’s realised that the deadline is slipping. Often, if it gets near the end of a project and there are schedule problems, more resource gets allocated to the project – but what with the man-month interchangeability problem, and the time it takes to get new people up to speed, such actions will only ever make things worse. It’s the old ‘using petrol to put out a fire’ theme, which is where we get Brooks’ Law:
Adding manpower to a late software project makes it later.
Conceptual Integrity
Conceptual integrity within a system is about keeping the way it works coherent – following a core set of principles that apply throughout the program, from a user’s perspective. Brooks states that conceptual integrity is the most important aspect of systems design – it’s more important to have a coherent model than extra functionality. This is difficult to achieve in software, which is usually designed and nearly always built by multiple people.
Cathedrals created over many generations have similar issues – often different parts are of different styles, and while each may be magnificent, the overall effect is jarring. Reims is the counter-example, where several generations have stuck to the original concept, sacrificing some of their own ideas to keep the overall design ideas intact. Conceptual Integrity can be achieved by having a single (or very few) Architects, who share a single philosophy for how the system works.
When Brooks talks of an architect, he doesn’t mean a technical architect as I would think, more a ‘user experience’ manager responsible for the UI and the user’s mental model of the software. The architect works on behalf of the user, making the most of the available systems to get the most benefit for the customer. This is an interesting concept and not something I’ve seen much of – typically, on the medium-sized business application I’ve seen, there is no-one really considering the user experience. The BAs will collect and define the user requirements, but will have little input into the actual interface of the system, which is usually left up to the developers.
I would think this may be more of an issue for larger systems, but Brooks states he’d have one architect role defined in teams as small as four, so it’s definitely something I’ll try to look further into in future projects.
Second System Effect
Another of the many phrases you’ve probably heard that comes from this book is the ‘second system effect’. The idea is that the second system anyone designs is the most dangerous and most likely to fail. In the first, ‘he knows he doesn’t know what he’s doing, so he does it carefully and with great restraint’. But all of those good ideas he has are saved up, and all piled into the second system.
Accepting Change
The thing that surprised me the most in TMMM was the recognition back in ’75 of the need to adapt to requirements change as the project progresses. Although development was all done in a waterfall fashion, the language used to describe dealing with change sounds like something straight out of an agile book:
‘The first step is to accept the fact of change as a way of life, rather than an untoward and annoying exception. […] both the actual need and the user’s perception of that need will change as systems are built, tested and used.’
Brooks gives advice on designing for change, including modularisation, well-defined interfaces, and good documentation of these. Another piece of advice is to always Throw One Away – with new types of system, you are still working out the best way of tackling problems, so you should plan to use the first as a prototype and not use it at all in production.
The book explains that we need to be prepared for change, because as well as technical and business changes, the users will not know exactly what they want until they try the system – this is one of the main driving forces behind agile/iterative development, which Brooks now recognises as a better way of developing software. In a recent look back at the original articles, Brooks states:
‘The waterfall model, which was the way most people thought about software projects in 1975, unfortunately got enshrined into [the DoD specification] for all military software. This ensured it’s survival well past the time when most thoughtful practitioners had recognized it’s inadequacy and abandoned it. Fortunately, the DoD has since begun to see the light.’
Software Maintenance
When software is maintained, unlike maintenance of things such as mechanical or electrical systems, usually this ‘maintenance’ involves adding extra functionality. The problem is, this new functionality has a good chance (a reported 20-50%) of introducing new bugs – ‘Two steps forward, one step back’. In addition, bug fixes that are applied sometimes fix the local problem, but go on to create further problems in other parts of the system.
A study of OS releases showed that as time goes on, structure gets broken and entropy increases in the system, and more and more time is spent fixing the defects introduced by the releases. Eventually, adding new functionality is not worth the effort and the system must be replaced. As Brooks succinctly puts it,
‘Program maintenance is an entropy-increasing process, and even its most skillful execution merely delays the subsidence of the system into unfixable obsolescence.’
Project Slippage
Most projects don’t suffer a catastrophic failure, they get slowly later as time goes on. ‘How does a project get to be a year late? …One day at a time.’
Brooks suggests one way to battle this is to have very specific, unambiguous milestones. People don’t like to give bad news, and if something has a fuzzy meaning, it’s all to easy to fool yourself and your manager that things are going Ok. Studies show that people gradually reduce over-estimates as time goes on, but underestimates stay low until around three weeks before the deadline.
Brooks talks about how small delays often get ignored, but as soon as you start accepting these delays, the whole project can slip. You need to stay on top of things – the team needs to have ‘hustle’, as baseball coaches would say.
PERT charts (apparently like GANTT charts today) can be useful to identify the slippages that really are problematic as they’re on the critical path.
No Silver Bullet (1986)
This being the anniversary edition, there are an extra couple of sections, including this well-known essay from 1986. The basic precept is this: there will be no magnitude (10x) increase in development productivity in the next 10 years. One of the core concepts here is that there are two kinds of complexity in a system – essential and accidental (or incidental). Essential is core to the problem and incidental is the complexity caused by our way of programming it.
The central argument is that unless the incidental complexity accounts for over 9/10 of the overall complexity, even shrinking it to zero wouldn’t increase productivity tenfold. Brooks thinks it accounts for way less that that, as the biggest problems (low-level languages, batch debugging, etc) have now been removed, so improvements will be fairly small from here on out. As such, we need to focus on attacking the essential complexity of software. To do this, Brooks suggests:
- Buying instead of building software
- Rapid prototyping in planned iterations to help get the requirements right
- Growing systems organically, adding functionality as they are used
These are all forward-looking views for ’86 which have largely been borne out in the intervening years. The last two points are related to the problems of capturing and identifying requirements properly, which Brooks sees as being the most difficult area, which is where we can make the most improvements:
‘The hardest single part of building a software system is deciding precisely what to build.’
30 Years On
At the end of the book, Brooks looks back on the assumptions and ideas presented those many years ago. Most have proved to be correct, although a few of them are based on Waterfall development, and while they are correct in terms of that approach, iterative development has rendered some of those ideas redundant.
The book ends on something of a high, with Brooks describing how when he finished college in the 50s, he could read all computer journals and conference proceedings. As time has gone on, he’s had to kiss goodbye to sub-discipline after sub-discipline, because there’s so much to know. So much opportunity for learning, research, and thought – ‘What a marvellous predicament! Not only is the end not in sight, the pace is not slackening. We have many future joys.’
Amen.
Stack Overflow
