On Design for Testability

Sunday, April 18th, 2010.

keeping balance Almost at every conference, event, training, or consulting engagement someone asks for my opinion on the whole design for testability thing. I’m not quite sure why I haven’t blogged on this topic, especially at the time that a lot of the other bloggers were weighing in, but better late than never.

Before getting into that, I want to start with a slightly broader scope of discussion.

You see, I get asked about “best practices” on all sorts of things. And I try not to be the kind of consultant that responds with “it depends”, but the context of the question often makes the answer irrelevant. And the unspoken context of a best-practice question is:

Given infinite time and budget

The biggest problem that I see with well-intentioned, best-practices-following developers and architects is that they don’t ask the question “is this the right thing for us to be focusing on right now?” Understandably, that is a difficult question to answer – but it needs to be asked, since you don’t have infinite time or budget to do everything according to best practices (assuming those even exist).

About testing

The biggest issue I have with the “design for testability” topic is the extremely narrow view it takes of the word “testability”, usually in the form of more code written by a developer which invokes the production code of the system, also known as “unit tests”.

There are many different kinds of testing – unit, integration, functional, load, performance, exploratory, etc… where some may be automated and others not. Should we not discuss what “design for testability” means for not-just-unit-testing?

And what’s the point of testing anyway?

It’s not to find bugs.

Research has shown that testing (of all kinds) is not the most effective way of finding bugs. I don’t have the reference handy but I’m pretty sure that it’s from Alistair Cockburn’s work. Code reviews are (on average) about 60% more effective.

Don’t get me wrong – testing can provide indications that the software has bugs in it, but not necessarily where in the code those bugs are.

The purpose of testing is to provide quantitative and qualitative information about the system that can help various stakeholders in their decision-making processes. The relevance of that information indicates the quality of the testing. Here are some examples:

The system supports 100 concurrent users, with the expected user-type distribution (X% role A, Y% role B, etc), performing expected use-case distributions, and collaboration scenarios.
Time to proficiency for new users in role A is expected to be 3 days
Alternate #2 of use case #12 fails on step #3

As you can see, the relevance of the above information is dependent on what decisions the various stakeholders need to make. The bullet on load can help us decide if more machines are needed or if developers need to tune the performance of the systems. The bullet on time to proficiency can help us decide if larger investment in usability is required. Information like the last bullet can be used in conjunction with the first two to decide on the timing and type of a release.

The timeliness of this relevant information is critical to the success of a project.

Choosing which and how much of the various testing activities to perform when is something that needs to be revisited several times throughout the lifetime of a project, taking into account the current risks (threats and probabilities) and time and resource investment to mitigate them.

Let me reiterate – we’re not going to have enough time to do everything.

On iterations

If the only part of your organization that is doing iterations are your developers, you’re not agile.

In order to capitalize on the information that testers are providing, you need them in your iterations.

The same goes for the other roles involved in the project – business analysts, DBAs, sysadmins, etc.

I know that 99% of organizations aren’t structured in a way to do this.

I never said doing this would be easy.

On design

Figuring out what kind of design and how much to do when is just as important, and just as hard. Design for testability is one part of that, but not the only one, or necessarily the most important one at any point of time.

Within that design for testability topic is the “design for unit-testing” sub-topic which seems to be the popular one. Before getting into the design aspects of it, let’s take a closer look at the unit-testing side of things.

On unit-testing

The assumption is that having more unit tests will lead to a code-base with less bugs, thus requiring shorter time to get the system into production, which will pay back the time it took to write those unit tests to begin with.

In practice, what tends to happen is that as development progresses, testing code breaks as the structure of the production code changes. Now one of two things happens – either the testing code is removed or rewritten. In either case, we didn’t get the return on investment we expected on the first bit of testing code. Unfortunately, rare is the case where the relevant people in the organization understand why, resulting in the same situation repeating itself over and over again.

Those projects would have been better off without unit testing, though the organization as a whole might have used those experiences to learn and improve. It’s been my experience that if the organization wasn’t conscious enough in the context of the project to notice the situation, it is unlikely to do so at higher levels.

On fragile unit tests

The reason that a unit test ends up being rewritten (or removed) is that its code was coupled to the production code in such a way that it broke when the production code changed. This tendency to break (fragility) is a critical property of a unit test. A fragile unit test will slow down a developer doing work on some existing code – it actually makes the system less maintainable.

For a unit test code to be stable (not fragile) it needs to be coupled to stable properties of the production code. The question of whether the production code is designed in such a way that it has stable properties – is a design question. Is it a unit? If not, you will not be able to write a unit-test against it.

And anyway, who said that every class is a unit, or should be a unit? Domain models (when done right) are good examples of a unit, yet the classes that make them up may not be units. Unit-testing should only be attempted with things which are units.

I think too much weight is put on whether a dependency of a class is a concrete or interface type, and not nearly enough on the nature of the dependency. I wouldn’t blame the hammer for pounding my thumb, and by the same token I think that blame should not be directed towards tools like those from TypeMock.

On tools

There is so much more depth to both design and testability that needs to be more broadly understood. No tool has yet been created to handle either design or testing in such a way that humans can give up responsibility for the outcome.

Over the years I’ve noticed that tools are most significant when used by skilled practitioners, which makes sense in retrospect. Giving a novice carpenter a laser-guided saw probably won’t significantly change the outcome of their work. Ultimately, the skilled practitioners are the ones that create tools – not the novices. And no tool, no matter how advanced, will make a novice perform at levels like the skilled practitioner.

In the case of a project too big for a single skilled practitioner to complete in the time required (or at all), the balance of importance shifts away from tools to the project management topics described above.

In summary

I hope that this post has shed some light on the context in which decisions with respect to testing need to be made. Design is one activity that can support certain kinds of testing, but not the only one, or even the most important one for the given type of testing necessary at that time in the project.

Design is hard. Project management is hard. Testing is hard.

Getting the right mix of people that together have enough experience and skills in these activities isn’t easy.

Don’t expect that sprinkling some interfaces in your code base will be enough.
That doesn’t count much in the way of design, just as writing code in a testing namespace doesn’t count much in the way of testability.

Looking forward to hearing your comments.

If you liked this article, you might also like articles in these categories:

If you've got a minute, you might enjoy taking a look at some of my best articles.
I've gone through the hundreds of articles I've written over the past 6 years and put together a list of the best ones as ranked by my 5000+ readers.
You won't be disappointed.

If you'd like to get new articles sent to you when they're published, it's easy and free.
Subscribe right here.

Follow me on Twitter @UdiDahan.

Something on your mind? Got a question? I'd be thrilled to hear it.
Leave a comment below or email me, whatever works for you.

20 Comments

Scott Bellware Says:
April 18th, 2010 at 1:54 pm
Great stuff, Udi. Always nice to see challenges to the status quo, or “orthodoxy”, as I’m want to call it.

That said, I’ve got a lot of experience in balancing internal testing with external testing and my own experience doesn’t jive with all that you’re reporting.

Ultimately, it comes to a matter of competence and learning, which you point out in conclusion. Any organization not learning from their practicing is going to end up with sub-par results on anything they do. Any organization creating fragility – whether designing a marketing strategy or designing a software unit is going to have pain.

I completely disagree with the iteration bias. We’ve begun to abandon iterations and optimize team organization and work management and design beyond what Scrum’s assertions can effectively hold up to. For me, this started more than two years ago, and my work management and organization management expectations are quite a bit different from what they used to be. This includes how we’re making use of testing and the meaning and purpose of testing and its effect on productivity. In the end, I don’t think “agile” has anything to do with this. “Agile” has become so watered down that it’s mostly meaningless at this point, and often an unproductive distraction except when rehabilitating very poorly managed software organization who are still trying to overly manufacturing organizational mechanics and values onto software development. Even then, I’d rather go straight to iterationless work management and control without stopping at the Scrum step.

Ultimately, doing things poorly usually means having poor results, and doing things goodly means having good results. This is the ultimate mitigator, and the fruit that most organizations fail to harvest whether in software projects or other aspects of business development and operations.

Basharat Wani Says:
April 18th, 2010 at 6:25 pm
Hey Udi

Good stuff, I have been reading your post since last few years, also many times I quote from your post.

Going through “On Design for Testability”, I see few things differently, I considered the Unit testing as “white-box” testing — the tests are performed with full knowledge of the source code. Ideally, developer would be able to run unit tests so that all the code paths would be executed as a result of the testing. In other words, the unit tests would need to create the conditions to go through each line of code to ensure all the code was operating correctly today and tomorrow.

Unit Test is very critical because it is one of the first testing efforts performed on the code and the earlier defects are detected, the easier they are to fix. Early bug-detection is also the most cost-effective methodology, earlier we catch them less it would cost and later they are found higher the cost.

Also when we say Unit-testing should only be attempted with things which are units (or components), this is very subjective and depends on how much capital we want to spend, but I would always prefer to have all the new code written to have unit tests coverage, as an Investment in Future.

Basharat

Scott Peterson Says:
April 18th, 2010 at 9:36 pm
Testable code is usually cohesive as well. Maybe that’s the real benefit to unit testing – it makes poor cohesion harder to ignore.

kris Says:
April 19th, 2010 at 1:55 am
hi Udi,

You’re missing a very important use case for unit tests (or component/integration tests): refactoring! after a year of production you’ll usually have gathered so much more insight into the business domain that you’ll have to refaactor either to implement new features or to improve the performance. having a suite of unit tests will help you achieve this task more quickly.

g,
kris

Chris Oldwood Says:
April 19th, 2010 at 7:44 am
“In either case, we didn’t get the return on investment we expected on the first bit of testing code.”

Your ROI is more than that – each time you run the tests you gain value. You either know sooner that you’ve broken something or have confidence that you hopefully didn’t. The tighter feedback loop *is* your ROI.

Szymon Kulec Says:
April 19th, 2010 at 1:28 pm
Considering domain models, or maybe parts bigger than a single class, as units, was also mentioned on the Ayende’s blog. I often include this kind of tests as unit tests, but in the majority, my tests still cover single classes. Don’t you think than wrapping the whole domain model is too much?

Have you ever informed your employer about possible paths for the project? For instance: ‘plenty of tests will cost you additional two week’, ‘without these we can finish the project on time’ and so on?

Take care
Szymon

Travis Laborde Says:
April 20th, 2010 at 7:09 am
I disagree strongly that there is an argument to be made against “Design for Testability” based on time and budget. It’s simply a matter of developer training and ability. As an old saying goes – “the opposite of testable code is… detestable code” 🙂

Designing for Testability means adhering to SRP, DRY, and loose coupling principles. Over just about any time span, these principles SAVE time and therefore budget. Even if you don’t write any tests at all, the benefit gained from designing your code in the “testable” way are immense.

udidahan Says:
April 21st, 2010 at 12:35 am
Scott Bellware,

Thanks for your comments – I think we’re very much on the same page. If a certain bias for iterations exists in the post, that is more of a reflection of the industry than what I believe is best. That being said, I do temper my guidance very much based on what an organization is willing and able to hear at a given time, so I may recommend iterations from time to time as the political realities dictate.

udidahan Says:
April 21st, 2010 at 12:41 am
Basharat,

Much of what you’re saying is “accepted truth” in the industry, but in my consulting I have seen a very different reality. Just like not everyone who writes code can write good code, not everyone who writes “unit tests” does that well.

The assumption that every unit test written is good does not hold up under scrutiny. In which case, it isn’t necessarily catching or preventing bugs – it’s just more code to maintain.

All I’m saying is for us to be more conscious of the investment choices we’re making, and following up with appropriate reflection from time to time on the return on those investments with real evidence rather than faith.

Thanks for your comments.

udidahan Says:
April 21st, 2010 at 12:46 am
Scott Peterson,

The term “testable code” is open to interpretation. If the test in question is just asserting that the code is implemented a certain way, “testable code” just means “more code” and is actually a violation of the DRY principle.

I agree that good code is often easy to test, but the inverse isn’t necessarily true – not all animals are dogs 🙂

This stuff isn’t easy, nor is it necessarily easy to teach.

udidahan Says:
April 21st, 2010 at 12:52 am
Kris,

The purpose of refactoring is to improve the design of the code while maintaining its behavior. It is unlikely that that is the highest business priority after a year of production. It is more likely that maintenance programmers will be fixing bugs and adding features – both of which change behavior, ergo not refactoring.

While I agree that a good suite of unit tests could help in refactoring a well designed code base, there is no inherent assurance that the unit tests are good, or that the code base well designed.

Hidden dependencies through shared databases and web services are often the places where design and unit tests break down and refactoring doesn’t necessarily help.

Cheers.

udidahan Says:
April 21st, 2010 at 12:55 am
Chris,

As I mentioned in the previous comment, poorly designed systems have all kinds of hidden dependencies that unit tests don’t target. Changes in one part of a system still can break other parts without unit tests going red. In which case, we’re operating under a false sense of security. Is that really such a good thing?

udidahan Says:
April 21st, 2010 at 12:59 am
Szymon,

I think that most people don’t divide up the system at the right level of granularity (services, business components) which results in domain models which are too large, in which case they’re not a unit either 🙂

When talking to clients about unit testing, I rarely get to the point of discussing the cost of *just* unit testing. Rather I focus on helping them find the right mix of the right kinds of testing throughout the lifecycle of the project – along with getting the right kinds of design decisions, requirements, etc…

Cheers.

udidahan Says:
April 21st, 2010 at 1:06 am
Travis,

Your statement “It’s simply a matter of developer training and ability” is almost right. I’d remove the word “simply” from that.

Is it easy to get developers with the right level of ability?
Is it easy to train those developers that are at lower levels of ability?
And what do we do in the meantime?

I’d say that “SRP, DRY, and loose coupling principles” fall under Good Design rather than Design for Testability. As I said in a previous comment, well designed code tends to be testable, but the inverse isn’t necessarily true.

There is lots of value to be gained from well designed systems – I agree. It’s just that we need to understand that that doesn’t happen by itself – hiring practices need to be changed, as do training practices, and more. It is a broader organizational issue than can be solved just by developers writing “unit tests”.

Zilvinas Says:
April 21st, 2010 at 2:55 am
Udi,

Great post. I was wondering if you would be able to recommend any books on testability issues.

Thanks.

Johannes Brodwall Says:
April 22nd, 2010 at 4:41 pm
It’s always sad to see people who assume that just because they haven’t been able to master something, others won’t get value from it.

True, it took me a few years to learn test-driven design to the point where I really could benefit from it. But once I and a few other senior developers had that experience, disseminating the skills to level where we experience benefit require only a few months of pair programming and occasional code dojos.

Once we have a part of our code properly covered with non-coupled tests, the added nimbleness gives us the confidence that we can design for today’s requirements and adapt the code to meet future needs. Since we practice incremental and iterative development, this is essential.

udidahan Says:
April 23rd, 2010 at 1:17 am
Zilvinas,

Read books written by testers for testers – James Bach is one of the good authors in that space.

udidahan Says:
April 23rd, 2010 at 1:22 am
Johannes,

On your statement:

“it took me a few years to learn test-driven design to the point where I really could benefit from it”

I’ve found that to be common. If Design-for-Testability advocates would include the fact that this is a long process involving deep skill acquisition, I wouldn’t have any issue with it.

What I don’t like is the message that if you have “unit tests” for your code, and your code makes use of interfaces, you will magically get all kinds of benefits.

Anyway, thanks for your comments.

Shrike Says:
May 17th, 2010 at 8:11 am
Udi,
it’s very interessing topic.
Some people say TDD is the only way to go and talk about testing in general from TDD’s point of view. That people tend to talk about “core” (as I call them) unit tests. Where “unit” is a class or method. They insist devs should write only unit-tests which are short, fast and have as few dependencies as possible.
I can agree such tests are nice thing to have. But the biggest problem I personaly can see here is that such unit-tests do not guarantee our whole system is correct. They guarantee a system’s parts are only correct.
So we have to have some integration (or as they’re often called “layered tests”) tests.
But if we have to have integration tests why not to make them automated. So we have automated integration tests which guarantee us that out system is ok.
And here the complex part for me : how to seperate effort onto writing tests between unit and integration tests? How to find right balance?
If I try to explain “core” unit-testing to some mid-experienced dev I’ll have difficulties. As I have to explain why he should point his efforts onto tests which do not fully guarantee correctness of the whole system.

udidahan Says:
May 29th, 2010 at 8:57 pm
Shrike,

As I said in the article, you must assume that there are some tests that weren’t written, in which case there is no guarantee of correctness. Striking a balanace between different kinds of tests is difficult (as you said) but is important – and no, not all tests need to be automated necessarily.