In one of Uncle Bob’s recent blog posts on the Single Responsibility Principle he uses the example of using people and organization boundaries as an indication of possible good software boundaries:

When you write a software module, you want to make sure that when changes are requested, those changes can only originate from a single person, or rather, a single tightly coupled group of people representing a single narrowly defined business function. You want to isolate your modules from the complexities of the organization as a whole, and design your systems such that each module is responsible (responds to) the needs of just that one business function.

This is something that often comes up when I teach people about service boundaries when it comes to SOA – organization boundaries are the most intuitive choice.

And, once up on a time, that intuition might have indeed held up.

Stepping back in time

In the age before computers, organizations had a very specific way of structuring themselves.

People who had to work closely together sat in close physical proximity to each other. Data that was required on an ongoing basis would be in file cabinets also physically co-located with the people using that data, and it would be structured in a way that was optimal for their specific purposes. All of this was due to the high cost of communicating with people farther away.

If you needed data from a different department, you had requisition it by filling out a special form, put it in your outbox, and then some guy from the mail room would pick it up, and physically schlep it to the right department, putting it in their inbox, and then someone there would get your data for you – putting it together with your original request, and then the mail guy would schlep it back. This inbox/outbox style of communication should ring a bell from the messaging patterns I talk about with NServiceBus.

As a result, different departments had to have very clearly delineated responsibilities with minimal overlap with each other. The organization just couldn’t function any other way.

And then a bunch of us geeks came along.

Enter the age of computers and networks

By introducing this technology, the cost of communication across large distances started falling – slowly at first, and then quite dramatically.

When anyone in an organization was able access data from anywhere in the blink of an eye, an interesting dynamic started to unfold. All of a sudden, the division of responsibility between departments wasn’t as critical as it was before. When an employee needed to do something, there wasn’t this “that isn’t our job, you need to go to so-and-so” reaction. Because things could be done instantly, that’s exactly what happened.

And then came the politics

By removing the cost of communication, it became possible for more power-hungry people in the organization to start making (or trying to make) decisions that they couldn’t have made before. The introduction of computers into an organization was heralded as a new way of doing business – that the old organizational boundaries were a relic that we should leave behind us.

And thus can the re-org (the first of many).

Responsibilities and people were shuffled around, managers vied for more power, and politics took its’ place as one of the driving forces in the company structure.

Nowadays, if you want a decision made in a company, there isn’t just one person who has the authority to sign off on it anymore. No, you need to have meetings – and more meetings, with people you never knew existed in the company, or why on earth they should have a say on how something is supposed to get done. But that is now our reality: endlessly partially overlapping responsibilities across the organization.

So, what of the Single Responsibility Principle

This just makes it that much harder to decide how to structure our software – there is no map with nice clean borders. We need to be able to see past the organizational dysfunction around us, possibly looking for how the company might have worked 100 years ago if everything was done by paper. While this might be possible in domains that have been around that long (like banking, shipping, etc) but even there, given the networked world we now live in, things that used to be done entirely within a single company are now spread across many different entities taking part in transnational value networks.

In short – it’s freakin’ hard.

But it’s still important.

Just don’t buy too deeply into the idea that by getting the responsibilities of your software right, that you will somehow reduce the impact that all of that business dysfunction has on you as a software developer. Part of the maturation process for a company is cleaning up its’ business processes in parallel to cleaning up its’ software processes.

The good news is that you’ll always have a job 🙂

There has been some discussion online recently about the issue of estimates in software development, specifically under a meme called #NoEstimates.

This came up when I was in London for the DevWeek conference at the speaker dinner with Austin Bingham, Rob Smallshire, and Allen Holub and I wanted to share some of the ideas that came up, as well as some of my personal opinions on the matter.

Context matters

When you’re in an organization that is continuously developing and evolving a product, platform, or suite, your context is quite different than when you’re working on a project either for an external client or an internal one.

In short: product vs. project.

While in both contexts you’ll want frequent releases, the main difference is that a project is meant to achieve a certain state of “done-ness” in a bound period of time. A product is not ever meant to be “done” in that way. A successful product is one that continues to evolve over time, with that success (arguably) resulting in more resources being dedicated to its development.

If we were to zoom out our scope beyond that of the project, we’d probably see certain product-ish qualities at the level of a client’s entire IT environment – no state of “done-ness” and similar consequences of success.

Where estimates are needed

Whether you’re in a product development organization or the CIO of an enterprise, there are a certain number of features/projects (FP) that people in “the business” want done. Let’s assume that each FP has a certain amount of business value that its implementation would result in and that that value is known in advance.

Sidebar: Clearly, the business value of any feature or project can not be known with much certainty in advance of it being implemented. Still, for the purpose of keeping the analysis simple for now, let’s table this issue for a bit.

While you might think it’s reasonable to perform the work on these FPs in order of decreasing value, that is mistaking revenue for profit/return-on-investment (ROI).

In other words, we need to know roughly how much each of them costs to be able to calculate its predicted ROI (value – cost).

Only then can we decide in which order to do the work.

So, we need somebody technical to give an estimate.

High-level estimates

In this context, sometimes it’s enough to provide 3-4 buckets describing the amount of work – I like the approach of using shirt sizes: S, M, L, XL.

This can help the organization decide quickly to charter the development of the FPs with very high value and very low cost. These low hanging fruit are great for getting started, but when you’re done with them, then you have to decide between a bunch of FPs whose predicted ROI are all very close to each other.

Before doing that (!), it’s important that no projects with an XL size are fully chartered as is – no matter what the value.

When a technical person gives an XL estimate, that means “this is so big, I really have no idea how long it’ll take”. The variance can be huge here – in some cases, they might not even be certain if the request is doable without being given some time to do additional research first. And that is exactly why you need to carve out a certain chunk of time and resources for doing that research.

“But you don’t understand, Udi! We need this done ASAP.”

Believe me when I say that that ship has already sailed.

There’s a pretty good chance that before that extra-large FP is half-done, so much time will have gone by that business priorities will have shifted. Unfortunately, by that time so many resources will have been invested in it that nobody will have the guts (or political capital) to pull the plug on it. The best people will start leaving – sometimes to other FPs but, more often, the company as well. Even when these nightmarishly large missions are eventually done – they end up being something of a Pyrrhic victory.

So, what to do?

Well, beyond financing a certain amount of directed and scoped research and development to get a better handle on that beast, let me suggest this:

Enter the “Lean Startup”

If you haven’t heard about it yet, and regardless of whether you’re actually working at a startup (I’d say that it is even more important for large organization), you need to check out the Lean Startup.

While I won’t be able to do justice to it here, let me use this admittedly gross oversimplification:

You are mistaken about the predicted business value.

Really.

So what you need to do is to start applying the scientific method – a series of experiments in which you are looking for proof to validate your hypothesis about the predicted value, where the outcome of one experiment is used to formulate another hypothesis to be tested in the next experiment.

Let me say this differently – until you are as rigorous in evaluating the predicted value of a given initiative as you are in estimating its cost, with that rigor increasing with the size of the initiative, you have no business starting to work on it.

And this is what’s missing from most of the software development world.

Portfolio management

I don’t really care what you call it, but the portfolio of potential investments and the risk analysis around them needs to be handled properly.

Based on the language I’ve chosen, you can see parallels to the world of finance. Now, before your mind starts going to the news of all the shady crap that’s been going on in the world of finance, understand that there are both positive and negative outliers in every domain.

That being said, I won’t point you to books on finance (at least, not for starters).

I suggest reading Manage your project portfolio, by Johanna Rothman (one of my favorite people in the world). She also has a couple of blogs, and that can help you get started.

This has already gotten quite long, so I’ll skip a bit and talk about what you should do if you’re “just” a developer being asked to give an estimate.

How to give a good fine-grained estimate

Here’s the format to use:

This will take a team of N between T1 and T2 and I am P% confident in that range, with the following assumptions (1, 2, 3), and most importantly, that the team does not work on anything else during that period of time.

Let me repeat that important bit again:

Assuming the team does not work on anything else.

Most of the kinds of people who ask for estimates aren’t going to like that kind of answer. You may get some pushback, “don’t be clever – can this be done by the end of the year, or not?”. In other words, they’re asking you for a commitment – not an estimate. This is common with certain types of dysfunctional organizations – the project management people are only measured on adherence to schedule, not on whether the system solves the right problem. You, the developer, want to make sure you’re solving the right problem – you want to be Agile. But that’s not the point.

Let me repeat – Agile is not the point.

If the organization around you is dysfunctional, be smart – don’t try to be right. Solving the right business problem is the right thing to do, but often it’s not smart thing to do. If you don’t like having to deal with this kind of organizational politics, you had better find yourself a different organization – otherwise, you had better be smart.

How to receive good estimates

Assuming you’re a team lead, project manager, product owner, or something similar, here’s how to respond when someone gives you an estimate in the above format.

1. If P is subjectively high enough and T2 is subjectively low enough, give them the green light.
2. If P isn’t high enough, or the range of T1 to T2 is too broad:
   1. If you have previously given them time to do research (RT), double the value of RT.
   2. Make sure they are only doing research during this time (no other development).
   3. The purpose of the research is to increase P, or decrease the range of T1-T2
   4. When receiving the new estimate, go back to the beginning.

To be clear, “research” does not mean navel-gazing. It can and often will involve writing a bunch of code as well as on figuring out what the requirements should have been in the first place.

“But Udi, won’t this end up wasting time that could be better used on actually building the system?”

No.

In reality, this will end up paying back all the time that should have been spent up-stream on portfolio management and requirements analysis activities.

Which brings me to…

Working software over comprehensive documentation

Sometimes, writing high-quality working software is the last thing you want to do.
Specifically, the earlier you are in the project, the less likely that you should be focusing on code.

Often, the most cost-effective thing to do is some rapid-prototyping like creating UI mock-ups to verify what the system should really do.

This really deserves a blog post of its own (with thanks to Geert for finding the link), but suffice it to say that this is a skill-set all its own that should exist in every software development organization.

On other word on small startups – I often hear from people who are doing their own startup that want to do DDD, CQRS, SOA, and a bunch of other three-letter-acronyms they picked up on various blogs and books, because this time it’s up to them; this time they’re going to do it right.

No, no, no – stop that. Go read the Lean Startup. Then do it. And if you’re extremely lucky, you will be so successful that in a couple of years you will be in the position to rewrite the system – the only difference is that now you’ll REALLY know what the system is supposed to be.

In closing

OK – so that’s well over 1500 words I’ve spilled on the topic.

While the hashtag #NoEstimates makes a great soundbite, estimates are an important part of the information that needs to flow around the organization to help quantify and mitigate risk. Too often there are various organizational dysfunctions tangled up in the same areas as estimates which, I suppose, could give the impression that the estimates are to blame.

While I wish I could tell you that all you need to do is find an organization that doesn’t have any of these dysfunctions and all will be well for you, unfortunately there aren’t any organizations without them. Just like all families are dysfunctional in one way or another, so too are organizations. This is simply because each of us human beings is somewhat dysfunctional.

Well functioning organizations are made up of highly aware individuals – people who have become able to see and mitigate some of their personal issues, and thus can be patient as the people around them similarly work through their issues. Together, these individuals continuously create and adapt their working processes and systems to compensate for the various dysfunctions in the group.

Under these iteratively growing conditions of mutual trust, various kinds of estimates are performed at different levels and times and are just a normal part of communication and decision-making.

Please, don’t throw out the baby with the bath-water.

For much of the history of computers, programmers really only had one path to take – upward into management.

While you could go from Junior Programmer to Senior Programmer, sooner or later you were faced with the choice of becoming a Team Lead or having your career stagnate.

The primary difficult with becoming a team lead is that the skills that made you an excellent Senior Programmer didn’t really carry over to leading a team.

On leading teams

Much ink has been spilled (and keyboards been pounded) on this topic, so I’ll just give the common solution that is proposed to this issue – having a parallel technical career track to the traditional management track.

After being a senior programmer, developers can grow into architects and upwards. IBM, for example, has the title “Fellow” reserved for this ultimate level.

An IBM Fellow is an appointed position at IBM made by IBM’s CEO. Typically only 4 to 9 IBM Fellows are appointed each year in May or June. It is the highest honor a scientist, engineer, or programmer at IBM can achieve. —Wikipedia

All that’s well and good, but I have a feeling that something is still missing.

Why does it have to be either/or?

What if we allowed, nay – encouraged, developers to try both types of roles as they advance in their career?

After your first year as a senior programmer, you are then assigned to be a mentor to a junior programmer. You don’t assign them work, but take responsibility for some of their professional development. During this time, you also start learning what it takes to be a good team lead and developing your soft skills – yourself being mentored by a more senior team lead (probably not a good idea for it to be the team lead on the project you’re working on). From there, you take on a team lead role on a small-ish project leading 2-3 other developers.

During your time as a team lead, architects in the company work with you to deepen your technical knowledge of larger system concerns – grooming you for your next role: an architect. Your experience as a team lead gives you new found appreciation for managing technical risk.

Later, as an architect with developed soft-skills, you are now much more capable of getting teams to adopt your ideas and to want to do “the right thing”, rather than just deliver the project any way they know how. Even as you develop your expertise in various technological areas, the organization has an eye on bringing you back to being a team lead, this time on a larger project.

In praise of the zigzag

I think this has actually been happening more than just a little in our industry, although I believe it usually happens as people move from one company to another.

Is it possible that the limitations of the structures in their previous companies contributed to their choice to move to another company? Well, I wouldn’t discount it.

I don’t think companies should pigeonhole developers – either you’re an X or a Y.

Human beings thrive on variety and occasionally stretching out of their comfort zone.

I believe this model optimizes for people’s personal growth and can be tuned quite easily.

I think this is also very much in alignment with the Software Craftsmanship movement and can make it easier for companies to develop and hold on to the talent they’ve been lucky enough to hire in the first place.

In closing

While I believe this model is workable for a lot of the software industry, both for consulting companies and internal development organization, it’s clearly not going to be applicable for small startups. That being said, if the startup is successful and starts growing, it might not be such a bad idea to lay the groundwork for the team early on.

I don’t claim that this is the “best model” out there, and I haven’t tried it myself (yet), but I do believe that it has potential and would love to (re)spark the conversation about processes and structure that has seemed to die down under the Agile maxim of “Individuals and interactions over processes and tools” (though I believe that Agile as a whole *has* gotten us pointed in a better direction than where the industry was before).

What are your thoughts?

Write a comment or, even better, write something on your blog and spread the word!

We’ve all seen good ideas emerge in the software space – from objects, to components, to services, to domain models, and the *DD approaches. Yet, in most organizations, it is very hard for these ideas to get traction.

I’ve heard from countless developers and architects over the years about their frustration in getting everybody else to go along with them. “Can’t they see how much better [new approach] is over what we’re doing now?!” they ask, believing that things could and actually would be evaluated on their merits, especially in a rational field like IT.

The usual explanation I give has a couple of parts.

Conway’s Law

In 1968 Melvin Conway penned what later became known as Conway’s Law which stated:

“organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.”

An important corollary of that law is that if you wish to have a significant impact on the design of a system, you would need to have a similarly significant impact on the communication structure of the organization making that system.

The main problem is that the people that tend to be pushing for DDD, IoC, CQRS, SOA, etc are usually not as strong when it comes to the soft skills that are so necessary for bringing about organizational change. The thing is that, at a minimum, these types of changes take 3 to 5 years so it really takes a long-term commitment, both from the individual and the organization.

On the rationality of people in IT

First of all, people are a whole lot less rational than they’d like to believe – or that they’d like other people to notice. In fact, people will go to great lengths to maintain the appearance of consistency and rationality, even at the cost of harm to themselves. How’s that for irrational?

Don’t take my word for it – there’s a great book on the topic: Predictably Irrational: The Hidden Forces That Shape Our Decisions. The somewhat scary thing about it is that not only are we irrational beings, but that that irrationality can be predicted and, yes, even manipulated.

Once you can understand that the people you’re trying to convince aren’t Vulcan, you have a much better chance of being effective. I’d say that, for myself, understanding my own modes of irrationality increased my effectiveness as well, and made me quite a bit happier in life too.

Why you need to bring in a consultant

This isn’t me hawking my wares – believe me, I’m busy enough as it is, but let me know when this starts to sound familiar to you.

There’s a problem in your organization – could be that you’re not delivering software fast enough, high enough quality, whatever. Suffice it to say that Management isn’t happy. You’ve been living this pain for a while and know exactly what the source of the problem is (more often than not, management has at least a hand if not a whole arm in it). You come up with some recommendations, bring them to the higher-ups, but ultimately are ignored, dismissed, or don’t even get into the room.

Some time later, management brings in a Consultant (that’s right, with a capital ‘C’) who is there to figure out what’s wrong and come up with recommendations. In some cases, especially in larger organizations, they bring in a whole bunch of them from a brand name like McKinsey or Ernst & Young.

If these guys are smart, they listen to you, ultimately presenting your analysis and recommendations to management. Of course, those higher-ups are in awe of how quickly these guys were able to understand the inner workings of their organization. That awe lends instant credibility to their recommendations which are then adopted and given powerful political backing.

And you’re sitting there thinking, “but… but… but that’s what I was saying!!”.

It’s not the message – it’s the messenger.

Let me put it another way, explained from the perspective of management – we’re having problems, you work here, ergo you’re part of the problem. Also, you don’t make that much money (compared to management), so how smart could you be? Those brand-name consultants, well, they cost a LOT, so they MUST be good (good enough to know not to work here too).

Therefore the more the consultant costs, the more likely management is to listen, which ultimately creates the conditions for success, which makes the change happen, which proves to management that they were right to bring in an expensive consultant. A vicious (or virtuous) cycle – depending on how you look at it.

Now, it doesn’t always work this way, but it does often enough to perpetuate management’s world view.

In Closing

I do hope your organization and its leaders aren’t trapped in this kind of dysfunction, but if they are, know that you’re not alone and that you can get help – either via consulting or in some books:

Some good books include Made to Stick: Why Some Ideas Survive and Others Die and the grandfather of the field: How to Win Friends & Influence People. There are countless others and there isn’t any right place to start – the most important thing to do is to start.

It’s been over 40 years since Melvin Conway’s observation and, as an industry, we’re still relearning these things – usually through the school of hard knocks. But there is an upside here – I’m pretty sure that, knowing these patterns, you could pick up on some signals during the interviewing process and find a company that’s outgrown many of these issues – one that would be able to have more meritocratic discussions on technical choices.

In the worst case, you could become a consultant and make a living off of all this irrationality 🙂

There’s a pattern I’ve seen with companies that are starting on a large green-field project that gets them into trouble later on that I wanted to call out because the pitfall is quite subtle.

It’s something I touched on in my presentation last week about Balancing Architecture & Agile at the Agile Practitioners group in Israel but didn’t have as much time to get into.

Agile

You see, at the beginning of these projects, often our users or customers don’t exactly know what they want. Even if you go through some process of doing mock-ups, sometimes users can’t really know what they want until they can interact with working software.

This is one of the advantages of an agile approach that focuses on getting working software in the hands of users very early. It allows us to mitigate the risk of us building the wrong system.

Pitfalls

The mistake here is believing that this needs to be done in a production-ready manner, using all of the super-scalability and highly reliable infrastructural elements in our target deployment environment.

While there is value in building real functionality on top of your production infrastructure to learn more about the challenges in doing that and in that process, refining the API of your platform, baking more capabilities into it, etc, the problem is that you are slowing down the important learning of what system needs to be built.

Decouple those two things.

Build one to throw away.

Sometimes, the way to be most effective is to be somewhat inefficient.

Once you’ve gotten your users to the point that they can point at a piece of working software and say, “Yes – build me that”, then you can go about building it the right way on top of your production platform.

Let me go a step further, just to be crystal clear.

Do not unit-test that code. Do not use TDD, DDD, CQRS, Event Sourcing, SOA, etc.

Just get it done.

Architecture

You also want to have a good understanding of the use cases in your system before you make too many big architectural decisions (where that might even be just 1 decision, but you only find out after the fact). Some use cases are architecturally significant – many aren’t. This falls under the “just enough architecture up-front”, but is really very dependent on the experience and skills of the people on your team so I really can’t give you anything more prescriptive.

Of course, there’s another risk around this approach.

Organizational Dysfunction

In some organizations, there isn’t enough of an understanding or appreciation of the difference between production-ready software and a prototype. I think we’ve all been through this at one time or another. If users see “the software” works, we might be forced to deploy the prototype – all our protests for naught.

If this is the environment you’re in, don’t worry about it.

Seriously – don’t.

Growing up

You can explain to a child that fire is hot, and that they will get burned if they stick their hand in it, but all of that is theoretical – it doesn’t really sink in, because they haven’t yet had a visceral experience of “hot”.

If the organization you’re in isn’t that mature yet, they’re going to have to get burned a couple of times until they learn. The only thing you can do as a responsible adult is to *try* to get them started with smaller burns so that they’ll learn the lessons with the minimal amount of pain.

Given that the people doing the learning are higher up the food-chain, you really don’t have much influence. Accept that and move on. If necessary, cover your assets – write an email containing the “meeting minutes” in which you describe the concerns that you raised and that the decision was made against your recommendations – but phrase it gently. Depending on the state of the economy and your network, it might be a good time to start job-hunting.

In closing

I said it before but it bears repeating:

Efficiency is secondary to effectiveness.

You want to make sure your headed in the right direction before putting the pedal to the metal.

Rework is to be accepted and, yes, even embraced.

There’s even a book about it.

Have fun.

Almost at every conference, event, training, or consulting engagement someone asks for my opinion on the whole design for testability thing. I’m not quite sure why I haven’t blogged on this topic, especially at the time that a lot of the other bloggers were weighing in, but better late than never.

Before getting into that, I want to start with a slightly broader scope of discussion.

You see, I get asked about “best practices” on all sorts of things. And I try not to be the kind of consultant that responds with “it depends”, but the context of the question often makes the answer irrelevant. And the unspoken context of a best-practice question is:

Given infinite time and budget

The biggest problem that I see with well-intentioned, best-practices-following developers and architects is that they don’t ask the question “is this the right thing for us to be focusing on right now?” Understandably, that is a difficult question to answer – but it needs to be asked, since you don’t have infinite time or budget to do everything according to best practices (assuming those even exist).

About testing

The biggest issue I have with the “design for testability” topic is the extremely narrow view it takes of the word “testability”, usually in the form of more code written by a developer which invokes the production code of the system, also known as “unit tests”.

There are many different kinds of testing – unit, integration, functional, load, performance, exploratory, etc… where some may be automated and others not. Should we not discuss what “design for testability” means for not-just-unit-testing?

And what’s the point of testing anyway?

It’s not to find bugs.

Research has shown that testing (of all kinds) is not the most effective way of finding bugs. I don’t have the reference handy but I’m pretty sure that it’s from Alistair Cockburn’s work. Code reviews are (on average) about 60% more effective.

Don’t get me wrong – testing can provide indications that the software has bugs in it, but not necessarily where in the code those bugs are.

The purpose of testing is to provide quantitative and qualitative information about the system that can help various stakeholders in their decision-making processes. The relevance of that information indicates the quality of the testing. Here are some examples:

• The system supports 100 concurrent users, with the expected user-type distribution (X% role A, Y% role B, etc), performing expected use-case distributions, and collaboration scenarios.
• Time to proficiency for new users in role A is expected to be 3 days
• Alternate #2 of use case #12 fails on step #3

As you can see, the relevance of the above information is dependent on what decisions the various stakeholders need to make. The bullet on load can help us decide if more machines are needed or if developers need to tune the performance of the systems. The bullet on time to proficiency can help us decide if larger investment in usability is required. Information like the last bullet can be used in conjunction with the first two to decide on the timing and type of a release.

The timeliness of this relevant information is critical to the success of a project.

Choosing which and how much of the various testing activities to perform when is something that needs to be revisited several times throughout the lifetime of a project, taking into account the current risks (threats and probabilities) and time and resource investment to mitigate them.

Let me reiterate – we’re not going to have enough time to do everything.

On iterations

If the only part of your organization that is doing iterations are your developers, you’re not agile.

In order to capitalize on the information that testers are providing, you need them in your iterations.

The same goes for the other roles involved in the project – business analysts, DBAs, sysadmins, etc.

I know that 99% of organizations aren’t structured in a way to do this.

I never said doing this would be easy.

On design

Figuring out what kind of design and how much to do when is just as important, and just as hard. Design for testability is one part of that, but not the only one, or necessarily the most important one at any point of time.

Within that design for testability topic is the “design for unit-testing” sub-topic which seems to be the popular one. Before getting into the design aspects of it, let’s take a closer look at the unit-testing side of things.

On unit-testing

The assumption is that having more unit tests will lead to a code-base with less bugs, thus requiring shorter time to get the system into production, which will pay back the time it took to write those unit tests to begin with.

In practice, what tends to happen is that as development progresses, testing code breaks as the structure of the production code changes. Now one of two things happens – either the testing code is removed or rewritten. In either case, we didn’t get the return on investment we expected on the first bit of testing code. Unfortunately, rare is the case where the relevant people in the organization understand why, resulting in the same situation repeating itself over and over again.

Those projects would have been better off without unit testing, though the organization as a whole might have used those experiences to learn and improve. It’s been my experience that if the organization wasn’t conscious enough in the context of the project to notice the situation, it is unlikely to do so at higher levels.

On fragile unit tests

The reason that a unit test ends up being rewritten (or removed) is that its code was coupled to the production code in such a way that it broke when the production code changed. This tendency to break (fragility) is a critical property of a unit test. A fragile unit test will slow down a developer doing work on some existing code – it actually makes the system less maintainable.

For a unit test code to be stable (not fragile) it needs to be coupled to stable properties of the production code. The question of whether the production code is designed in such a way that it has stable properties – is a design question. Is it a unit? If not, you will not be able to write a unit-test against it.

And anyway, who said that every class is a unit, or should be a unit? Domain models (when done right) are good examples of a unit, yet the classes that make them up may not be units. Unit-testing should only be attempted with things which are units.

I think too much weight is put on whether a dependency of a class is a concrete or interface type, and not nearly enough on the nature of the dependency. I wouldn’t blame the hammer for pounding my thumb, and by the same token I think that blame should not be directed towards tools like those from TypeMock.

On tools

There is so much more depth to both design and testability that needs to be more broadly understood. No tool has yet been created to handle either design or testing in such a way that humans can give up responsibility for the outcome.

Over the years I’ve noticed that tools are most significant when used by skilled practitioners, which makes sense in retrospect. Giving a novice carpenter a laser-guided saw probably won’t significantly change the outcome of their work. Ultimately, the skilled practitioners are the ones that create tools – not the novices. And no tool, no matter how advanced, will make a novice perform at levels like the skilled practitioner.

In the case of a project too big for a single skilled practitioner to complete in the time required (or at all), the balance of importance shifts away from tools to the project management topics described above.

In summary

I hope that this post has shed some light on the context in which decisions with respect to testing need to be made. Design is one activity that can support certain kinds of testing, but not the only one, or even the most important one for the given type of testing necessary at that time in the project.

Design is hard. Project management is hard. Testing is hard.

Getting the right mix of people that together have enough experience and skills in these activities isn’t easy.

Don’t expect that sprinkling some interfaces in your code base will be enough.
That doesn’t count much in the way of design, just as writing code in a testing namespace doesn’t count much in the way of testability.

Looking forward to hearing your comments.

Convention over configuration describes a style of development made popular by Ruby on Rails which has gained a great deal of traction in the .net ecosystem. After using frameworks designed in this way, I can say that the popularity is justified – it is much more pleasurable developing this way.

The thing is, when looking at this in light of the full software development lifecycle, there are signs that the waters run deeper than we might have originally thought.

Let’s take things one step at a time though…

What is it?

Wikipedia tells us:

“Convention over Configuration (aka Coding by convention) is a software design paradigm which seeks to decrease the number of decisions that developers need to make, gaining simplicity, but not necessarily losing flexibility. The phrase essentially means a developer only needs to specify unconventional aspects of the application.”

What this means is that frameworks built in this way have default implementations that can be swapped out if needed. So far so good.

For example…

In NServiceBus, there is an abstraction for how subscription data is stored and multiple implementations – one in-memory, another using a durable MSMQ queue, and a third which uses a database. The convention for that part of the system is that the MSMQ implementation will be used, unless something else is specified.

Developers wishing to specify a different implementation can specify the desired implementation in the container – either one that comes out of the box, or their own implementation of ISubscriptionStorage.

Things get more interesting when we consider the full lifecycle.

Lifecycle effects

When developers are in the early phases of writing a new service, they want to focus primarily on what the service does – its logic. They don’t want to muck around with MSMQ queues for storing subscriptions and would much rather use the in-memory storage.

As the service takes shape and the developers want to run the full service on their machine, possibly testing basic fault-tolerance behaviors – kill one service, see that the others get a timeout, bring the service back up, wanting it to maintain all the previous subscriptions.

Moving on from there, our developers want to take the same system they just tested on their machine and move it into a staging environment. There, they don’t want to use the MSMQ implementation for subscription storage, but rather the database implementation – as will be used in the production environment.

While it may not sound like a big deal – changing the code which specifies which implementation to use when moving from one environment to another, consider that on top of just subscription storage, there is logging (output to console, file, db?), saga persistence (in-memory, file-based DB, relational DB), and more.

It’s actually quite likely that something will get missed as we move the system between environments. Can there be a better way?

What if…

What if there was some way for the developer to express their intent to the system, and the system could change its conventions, without the developer having to change any code or configuration files?

You might compare this (in concept) to debug builds and release builds. Same code, same config, but the runtime behaves different between the two.

As I mulled over how we could capture that intent without any code or config changes, the solution that I kept coming to seemed too trivial at first, so I dismissed it. Yet, it was the simplest one that would work for console and WinForms applications, as well as windows services – command line arguments. The only thing is that I don’t think those are available for web applications.

But since we’re still in “what if” land, and I’m more thinking out loud here than providing workable solutions for tomorrow morning, let’s “what if” command line arguments worked for web apps too.

Command-Line Intent

Going back to our original scenario, when developers are working on the logic of the service, they run it using the generic NServiceBus host process, passing it the command line parameter /lite (or whatever). The host then automatically configures all the in-memory implementations.

As the system progresses, when the developer wants to run everything on their machine, they run the processes with /integration. The host then configures the appropriate implementations (MSMQ for subscription storage, SQLite for saga persistence, etc.

When the developers want to run the system in production, they could specify /production (or maybe that could be the default?), and the database backed implementations would be configured.

Imagine…

Imagine being able to move that fluidly from one environment to another. Not needing to pore over configuration files or startup script code which configures a zillion implementation details. Not needing to worry that as you moved the system to staging something would break.

Imagine short, frictionless iterations even for large scale systems.

Imagine – lifecycle-aware frameworks making all this imagination a reality.

In Closing

We’re not there yet – but we’re not that far either. The generic host we’re providing with NServiceBus 2.0 is now being extended to support exactly these scenarios.

It’s my hope that as more of us think about this challenge, we’ll come up with better solutions and more intelligent frameworks. Just as convention came to our rescue before, breaking us out of the pain of endless XML configuration, I hope this new family of lifecycle-aware frameworks will make the friction of moving a system through dev, test, staging, and production a thing of the past.

A worthy problem for us all to solve, don’t you think?

Any ideas on how to make it a reality?
Send them in – leave a comment below.

It’s one of the things that sometimes drives me mad about the YAGNI philosophy of Agile.

We need to stop throwing out the baby with the bath water.

Jay really liked that statement with relation to my previous post “Scalability – you wish you’re gonna need it“, so I thought I’d put up a logo for this movement. Anyone feeling like joining in, leave a comment, link, or whatever.

I understand that we don’t need to over-engineer everything, putting in every possible kind of extensibility point, so I accept that part of YAGNI. That is not a license to not think about the extensibility points you do need.

<Remarks>

This is a somewhat tongue-in-cheek post, and I do not want the pendulum to swing to far back the other way. But I do think it’s time it took a step back from the over-zealous “we’ll TDD our way there” thinking. Maybe Ron can pull it off. I’ve yet to see anyone else succeed.

</Remarks>

After reading Derek Hatchard’s post, The Art and War of Estimating and Scheduling Software, I wanted to follow up on my previous post on the topic, Don’t Trust Developers with Project Management. The problem lies with individualistic thinking.

Developers, and managers too for that matter, by and large are concerned with “productivity”. Developers want the latest tools and technologies so that they can churn out more code faster. Managers create schedules trying to get the maximum efficiency out of each one of their developers. They consider resource utilization and other terms that sound manager-ish.

Fact is, on medium to large sized projects, if you look at the studies you’ll find that developer productivity when measured as total lines of (non-blank) code of the system in production divided by the total number of developer days comes in roughly at 6. Maybe 7.

7 lines of code a day.

Let that sink in for a second.

I can hear the managers screaming already. OMFG, what were they doing all day long?! It takes, what, 10 minutes to put out 7 lines of code? An hour even, if it’s complicated recursive code and stuff. And they say they don’t like us micro-managing them?! Now we know why. It’s because they’re goofing off all day long.

Well, managers, that’s not really the way it goes. You see, you have to take into account the time it took to learn the technology, tools, frameworks, etc. Add to that the time of understanding the requirements, which is really sitting through boring meetings that don’t explain much. Finally, our poor developer actually gets to implement the requirement. Maybe run the system a couple of times, trying out the feature they implemented, and checking the code in.

Well, that’s actually the easy part. Now comes the part which kills most of the time. After a bunch of features have been developed by the team, the testers start banging away at it and find a bunch of bugs. Now the developer has to reverse-engineer some bizarre system behavior and figure out which part of the system is to blame. That involves usually some educated guessing (unless they’ve just joined the team and have been put in the bug-fixer role to “learn the system”, in which case it is thoroughly UNeducated guessing). They change some code, run the system, which looks like its been fixed, check the new code in, and close the bug.

But the bugs keep coming. And as the project progresses towards production, more and more of the developers time is spent looking through code and changing existing code, that actually writing new code.

And the larger the system, the more bugs. And I don’t mean that the number of bugs linearly increases with lines of code, or number of features. It’s probably closer to exponential. If it’s a mission critical system, the performance bugs will be taking an order of magnitude more time to fix than other bugs.

So, as you can see, getting a system into production is a team effort. It includes the developers and testers, of course, but also management, and the customer, and how they manage scope. This is kind of a “duh” statement, but we’re getting to the punch-line.

If getting a system into production involves the entire team, isn’t that obviously true for each feature too?

In which case, why are we asking just the developers to estimate the time it takes to get a feature “done”? Why are we trying so hard to measure their productivity?

I know why. It’s so we can get rid of the less productive ones and give bonuses to the more productive ones!

Back to the main issue. I don’t “trust” developer estimates because I need to see the team’s capability to put features in production. The involves all aspects, and often many team members, in some cases multiple developers going through the same code. This involves all overhead and cross team communication, sick days, etc. It’s also why I try to get multiple data points over time to understand the team’s velocity.

While I care about the quality of my developers, and testers, and everybody on my team and would like them to be able to estimate their work as best they can, I’ve got a project to put into production. And the best way I’ll know when it’ll go into production is by having data that’ll enable me to state to my management:

“Our team is finishing 20 feature-units a month, we’ve got 200 feature-units to go, so we’ll be done in around 10 months.”

If I’m busy micro-measuring each developers estimates, I won’t have the time to see the forest. By first taking a harsh look at the reality of what the team can do, I can start looking for ways to make it better. Maybe the bottleneck is between analysts and developers, maybe we’re seeing the same bugs regressing many times, but until we know where we are, we can’t run controlled experiments to see what makes us better.

Focusing on the individual developer, getting them the latest and greatest tools may be great for their morale, but it probably won’t make a bit of difference to their actual productivity.

Next time – what to do when management asks you what it’ll take to be done sooner.

Jeremy’s trying to answer some hard questions about agile. I wanted to tackle the issue of fixed-bids, since most of my clients work on those kinds of projects and I managed those projects full-time before becoming a consultant.

So, here’s the thing.

The only way to win on fixed-bid projects, is to bid low, and then rack up the change-requests. This is why people spend so much time documenting requirements, and then getting the client to sign off. It’s so they can prove that something is an actual change request, and thus they don’t have to do it. So, if the client wants to do whatever, they have to pay more money.

The problem is that it pisses off the client.

There’s another, subtler problem. It’s that clients get wise to this game, and front-load every possible requirement requesting total flexibility in everything.

This leads to another problem. We can’t bid low anymore.

Which leads to another problem. The client doesn’t have the budget to pay for the longer list of requirements.

Which leads us back to square one.

Fixed bids are a lose-lose proposition.

You see, if you bid rationally, taking into account the fact that some requirements will change, others will appear mid-way through, and so on, you’re bid will be significantly higher than the other guy who low-balled it. That means that the client will have a very hard time explaining to his management why he wants you to do the project.

So, the only way to win is for the client to realize this and game the system. This is sometimes a fine-line, possibly bordering on illegal when it comes to government contracts.

Once you have a client who understands that the fixed-bid is not in their interest, they will work collaboratively with you to get a reasonable system out the door within the given budget. There will be a lot of give-and-take but it can work. After a system goes into production successfully, it’s a lot easier to get management buy-in for the next version.

Fact is, upper management doesn’t really know all the specific requirements. So, if you don’t do them all, you’re OK, and so is your client.

In these circumstances, agile development is not only possible, but likely.

I know that it’s not really fixed-price, fixed-time, fixed-scope this way. But that’s what makes it successful 🙂