As I was reading a blog post on CQRS, Aggregate Roots, and Invariants here, I became aware of a mistake I’ve seen many developers make over the years and I thought I’d call it out real quick.

Superficial Invariants

Taken from the blog post mentioned above: “For example, an employee cannot take more annual leave than they have.”

This falls into the trap of applying mathematical thinking (which we developers possess in great quantities) to the business world. The business world isn’t that mathematical (in general), and tends to have many more shades of gray expressed as “business rules” which can, and do, change.

Rules – not invariants

Employees can’t take more annual leave than they have.
… unless their manager approves.
… but that’s only up to 2 days.
… unless their manager is a VP, and then it’s up to 5 days.
… and that negative balance will be deducted from next year’s leave.
… Oh, and if the employee leaves the company before then, then the value of those negative days will be deducted from their final paycheck.

Impact on your domain

First of all, I hope you see that this isn’t something that you would trivially implement on an Employee object.

If you read these rules more carefully, you’ll probably notice that they’re speaking about a long-running process.

First, there is a request for leave. Then there’s an approval (with certain rules) which may come sometime later. And the approval itself may not even end the process – if the balance becomes negative.

And, as you’ve probably heard me say before, you end up with sagas as your aggregate roots (see Race conditions don’t exist from 4 years ago).

And a word about Bounded Contexts

Notice that these rules don’t care very much about things like the employee’s name, phone number, email, etc. Similarly, logic that deals with that data probably doesn’t care about the number of days of leave an employee takes.

In other words, these sets of data and logic can be said to belong to different Sub-Domains (in DDD terminology).

As such, it can make sense to take the annual leave logic and put it in a bounded context separate from the one responsible for the contact info.

In closing

In many of the samples and blog posts I see online, an overly simplified problem domain is implemented showing how the given implementation technique would be applied.

The problem is that developers then use that implementation technique as a “cookie cutter”, trying to fit real-world requirements into it, and then end up making a pretty big mess.

The more you delve into real-world requirements of business domains, the less you’ll see of mathematical invariants (unless, maybe, you’re building a physics engine for a game or something) and the more you’ll see long-running processes unfolding in front of you.

Regardless of whether you use NServiceBus sagas or not, start looking at the world as dynamic long-running processes rather than static noun-centric entities.

With all the talk of CQRS, the area that doesn’t get enough treatment (in my opinion) is that of queries. Many are already beginning to understand the importance of task-based UIs and how that aligns to the underlying commands being sent, validated, and processed in the system as well as the benefits of messaging-centric infrastructure (like NServiceBus) for handling those commands reliably. When it comes to queries, though, it isn’t nearly as well understood what it means for a query to be “task based”.

Starting with CRUD

Let’s start with a traditional CRUD application and work our way out from there.

In these environments, we often see users asking us to build “excel-like” screens that allow them to view a set of data as well as sort, filter, and group that data along various axes. While we might not get this requirement right away, after some time users begin to ask us to allow them to “save” a certain “query” that they have set up, providing it some kind of name.

That, right there, is a task-based query and it is the beginning of deeper domain insight.

Pattern matching

Any time a user is repeatedly running the same query (this can be once a day or some other unit of time) there is some scenario that the business is trying to identify and is using that user as a pattern-matching engine to see if the data indicates that that scenario has occurred.

It’s quite common for us to get a requirement to add some field (often a boolean or enum) to an entity which defaults to some value and then see that same field used in filtering other queries. These measures are sometimes instituted as a temporary stop-gap while a larger feature is being implemented, though (as the saying goes) there is nothing more permanent than a temporary solution.

Where we developers go wrong

The thing is, many developers don’t notice these sorts of things happening because we don’t actually look at the kinds of queries users are running.

One excellent technique to better understand a domain is to sit down with your users while they’re working and ask them, “what made you run that query just now?”, “why that specific set of filters?”.

What I’ve noticed over the years is that our users find very creative ways to achieve their business objectives despite the limitations of the system that they’re working with. We developers ultimately see these as requirements, but they are better interpreted as workarounds.

I’ll talk some more about how a software development organization should deal with these workarounds in a future post, but I want to focus back in on the queries for now.

Oh, and don’t get me started on caching or NoSQL, not that I think that those tools don’t provide value – they do, but they’re only relevant once you know which business problem you’re solving and why.

Not all queries are created equal

Even before bringing up the questions I described in the previous section, any time you get query-centric requirements the first question to ask is “how often will the user be running this specific query?”.

If the answer is that the specific query will be run periodically (every day, week, etc), then drill deeper to see what pattern the user will be looking for in the data. If the person you’re talking to doesn’t know to answer that question, then go find someone who does. Every periodic query I’ve seen has some pattern behind it – and in my conversations with thousands of other developers over the years, I’ve seen that this is not just my personal experience.

But there is a case where a query does get run repeatedly without there being a pattern behind it.

I know this sounds like I’m contradicting myself, but the distinction is the word “specific” that I emphasized above.

There are certain users who behave very differently from other users – these users are often doing what I call research, i.e. the “I don’t know what I’m looking for but I’ll know it when I see it” people.

These researchers tend to repeatedly query the data in the system however they tend to run different queries all the time. This is the reason why traditional data warehouse type solutions don’t tend to work well for them. Data warehouses are optimized for running specific queries repeatedly.

Keeping the Single-Responsibility Principle in mind – we should not try to create a single query mechanism that will address these two very different and independently evolving needs.

And now on to Search

Search is a feature that is needed in many systems and whose complexity is greatly underestimated.

While the developer community has taken some decent strides in understanding that search needs to be treated differently from other queries, the common Lucene/Solr solutions that are applied are often overwhelmed by the size of the data set on which the business operates.

The problem is compounded by our user population being spoiled by Google – that simple little text box and voila, exactly what you’re looking for magically appears instantaneously. They don’t understand (or care) how much engineering effort went into making that “just work”.

Lucene and Solr work well when your data set isn’t too large, and then they become pretty useless as the quality of their results degrades. The thing is that many of us in IT tend to work on projects where we have an unrealistically small data set that we use to test the system and, at these volumes, it looks like our solutions work great. But if you have 20 million customers, do you think a full text search on “Smith” is going to find just the right one?

Larger data sets require a relevance engine – something that feeds off of what users do AFTER the query to influence the results of future queries. Did the user page to the next screen? That needs to be fed back in. Did they click on one of the results? That needs to be fed back in too. Did they go back to the search and do another similar search right after looking at a result – that should possibly undo the previous feedback.

And that’s just relevance for beginners.

You know what makes Google, you know, Google? It’s that they have this absolutely massive data set of what users do after the query that informs which results they return when. You probably don’t have that. That and search is/was their main business for many years – I’m betting that it’s not your main business.

You should discuss this with your stakeholders the next time they ask for search functionality in your system.

In closing

I know that the common CQRS talking points tell you to keep your queries simple, but that doesn’t mean that simple is easy.

It takes a fair bit of domain understanding to figure out what the queries in the system are supposed to be – what tasks users are trying to achieve through these queries. And even when you do reach this understanding, convincing various business stakeholders to change the design of the UI to reflect these insights is far from easy.

It often seems like the reasonable solution to give our users everything, to not limit them in any way, and then they’ll be able to do anything. What ends up happening is that our users end up drowning in a sea of data, unable to see the forest for the trees, ultimately resulting in the company not noticing important trends quickly enough (or at all) and therefore making poor business decisions.

Even if your company doesn’t believe itself to be in “Big Data” territory, I’d suggest talking with the people on the “front lines” just in case. Many of them will report feeling overwhelmed by the quantity of stuff (to use the correct scientific term) they need to deal with.

It’s not about Lucene, Solr, OData, SSRS, or any other technology.

It’s on you. Go get ’em.

Occasionally I’ll get questions from people who have been going down the CQRS path about why I’m so against data duplication. Aren’t the performance benefits of a denormalized view model justified, they ask. This is even more pronounced in geographically distributed systems where the “round-trip” may involve going outside your datacenter over a relatively slow link to another site.

CQRS

As his been said several times before by many others, it’s not the denormalized view model that defines CQRS.

One of the things that sometimes surprising people after going through my course is that in most cases you don’t need a denormalized view model, or at least, not the kind you think. Yes, that’s right: MOST cases.

But I don’t want to get too deep into the CQRS thing in this post – that can wait.

SOA

The big thing I’m against is raw business data being duplicated between services.

Data that can be expected to be accessible in multiple services includes things like identifiers, status information, and date-times. These date-times are used to anchor the status changes in time so that our system will behave correctly even if data/messages are processed out of order. Not all status information necessarily needs to be anchored in time explicitly – sometimes this can be implicit to the context of a given flow through the system.

For example, the Amazon.com checkout workflow.

In that flow, if you provide a shipping address that is in the US, you are presented with one set of options for shipping speed, whereas an international address will lead you to a different set of options.

Assuming that the address information of the customer and the shipping speed options are in different services, we need to propagate the status InternationalAddress(true/false) between these services in that same flow. In this case, there isn’t a need to explicitly anchor that status in time.

But what’s so bad about duplication of data between services?

The danger is that functionality ultimately follows raw business data.

You start with something small like having product prices in the catalog service, the order service, and the invoice service. Then, when you get requirements around supporting multiple currencies, you now need to implement that logic in multiple places, or create a shared library that all the services depend on.

These dependencies creep up on you slowly, tying your shoelaces together, gradually slowing down the pace of development, undermining the stability of your codebase where changes to one part of the system break other parts. It’s a slow death by a thousand cuts, and as a result nobody is exactly sure what big decision we made that caused everything to go so bad.

That’s the thing, it wasn’t viewed as a “big decision” but rather as just one “pragmatic choice” for that specific case. The first one excuses the second, which paves the way for third, and from that point on, it’s a “pattern” – how we do things around here; the proverbial slippery slope.

So what’s with the word “Replication” in the title of this post?

While data duplication between services is very dangerous, replication of business data WITHIN a service is perfectly alright.

Let’s get back into multi-site scenarios, like a retail chain that has a headquarters (HQ) and many stores. Prices are pushed out from the HQ and orders are pushed back from the stores according to some schedule.

We know that we can’t guarantee a perfect connection between all stores and the HQ at all times, therefore we copy the prices published from the HQ and store them locally in the store. Also, since we want to perform top-level analytics on the orders made at the various stores, that would be best done by having all of those orders copied locally at the HQ as well.

We should not view this movement of data from one physical location to another as duplication, but rather as replication done for performance reasons. If there were some magical always-on zero-latency network that existed, we wouldn’t need to do any of this replication.

And that’s just the thing – logical boundaries should not be impacted by these types of physical infrastructure choices (generally speaking). Since services are aligned with logical boundaries, we should expect to see them cross physical boundaries – this includes SYSTEM boundaries (since a system is really nothing more than a unit of deployment).

I know that you might be reading that and thinking “What!?” but there isn’t enough time to get into this in any more depth here. You can read some of my previous posts on the topic of SOA for more info here.

Cross-site integration without replication

There are some domains where sensitive data cannot be allowed to “rest” just anywhere. Let’s look at a healthcare environment where we’re integrating data from multiple hospitals and care providers. While all of these partners are interested in working together to make sure that patients get the best care, which means that they need to share their data with each other, they don’t want any of THEIR data to remain at any partner sites afterwards (and are quite adamant about this).

In these cases, the decision was made that performance is less important than data ownership. Personally, I don’t agree with this mindset. The fact that data is “at rest” in a location as opposed to “in flight” does not change ownership. It could be stored in an encrypted manner so that only a certain application could use it, resulting in the same overall effect, but this is an argument that I’ve never won.

People (as physical beings) put a great deal of emphasis on the physical locations of things. It’s understandable but quite counterproductive when dealing with the more abstract domain of software.

In closing

By virtue of the fact that we don’t duplicate raw business data between services, that means that the regular data structures inside a service already look very different from what they would have looked like in a traditional layered architecture with an ORM-persisted entity model.

In fact, you probably wouldn’t see very many relationships between entities at all.

Going beyond that, you probably wouldn’t see the same entities you had before. An Order wouldn’t exist the way you expect; addresses (billing and shipping) would be stored (indexed by OrderID) in one service whereas the shipping speed (also indexed by OrderId) would be in another, and the prices may well be in yet another.

It is in this manner that data does not end up being duplicated between services, but rather is composed by many services whether that is in the UI of one system, the print-outs down by a second system, or in the integration with 3rd parties done by a third system.

If performance needs to be improved, look at having these services replicate their data from one physical system to another – in-memory caching is one way of doing this, denormalized view models might be though of as another (until you realize there isn’t very much normalization within a service to begin with).

And a word from our sponsor 🙂

For those of you on “rewrite that big-ball-of-mud” projects looking to use these principles, I strongly suggest coming on one of my courses. The next one is in San Francisco and I’ve just opened up the registration for Miami.

For those of you on the other side of the Atlantic, the next courses will be in Stockholm in October and in London this December.

The schedule for next year is also coming together and it will include South Africa and Australia too.

Anyway, here’s what one attendee had to say after taking the course earlier this month:

I wanted to thank you for the excellent workshop in Toronto last week. I spent the better part of the weekend reflecting over what was presented, the insights we learned through the group exercises, and how my preconceptions of SOA have changed. By the end of the course, all the tidbits of (usually) rather ambiguous information that I’ve collected from various blogs, books, and other sources, finally coalesced into something more intelligible – one big A-HA moment if you will. Overall, I found the content of the workshop to be incredibly enlightening and it left me feeling invigorated and excited to learn more.
– Joel from Canada

Hope you’ll be able to make it.

If travel is out of the question for you, you can also look at get a recording of the course here.

One final thing

If your employer won’t foot the bill for these, please get in touch with me.
I wouldn’t want you not to be able to come just because you’re paying out of pocket.

There are very substantial discounts available.

Contact me.

Here’s a small quick tip that can help you improve the performance of important use cases in your systems. It doesn’t require very many changes to your code and can improve matters when your system is under load but won’t make much of a difference when you have capacity to spare.

This is something I talk about in the first day of my course when going through the fallacies of distributed computing – specifically fallacy #3 which talks about bandwidth.

What about bandwidth?

When it comes to network bandwidth in your datacenter, there’s a pretty good chance you’re still on gigabit ethernet. When most developers hear that prefix, “giga”, there is an instantaneous translation in their brain to “so much I don’t need to worry about it”.

The thing is that it’s GigaBIT, not GigaByte, so we’re talking about 128 MBps.

Also, keep in mind that hardly anybody programs at the level of ethernet, we’re several layers up the stack. You can expect roughly 40% the bandwidth of ethernet up in TCP land due to its collision detection, exponential backoff, etc. So that’s roughly 50MBps, not counting overhead for serialization (which can be very significant if it’s text-based like XML or JSON).

In practice, you might be getting something like 25MBps – definitely not so much that you don’t even need to think about it.

Everybody’s talking scalability in terms of number of servers and memory, storage, CPU per server – but what about the network? More importantly, what happens when (not if) you run out? Well, the latency of your calls increase – and that can be quite substantial.

Business Priority

And now we get to the crux of the matter.

Consider a “Customer” web service with a bunch of methods on it, including these two: GenerateTopCustomersByRegionReport and MarkCustomerAsFraud.

Now, your system is under significant load and there’s just enough bandwidth left for one call to make it across the network without timing out. Two users invoke the functionality above – one doing the report, the other doing the fraud. Should the fact that one user clicked the button a millisecond before the other mean that the MarkCustomerAsFraud should be delayed to the point of failure?

I’m fairly sure that if we asked a business stakeholder the answer would be a clear no.

While we could try asking the network engineers to give higher priority to that webmethod, let’s face it, that’s never going to happen. Since both methods are on the same web service, clients are bound to the address where that web service is hosted, regardless of which methods they call.

The problem with “the” network

If there’s one word I loathe in the English language, it’s the word “the”.

Such a small word, tacked on in front of so many other words that, without you even noticing it, traps your mind into thinking you can have only one of that thing.

The network.
The database.

But you CAN have more than one – it’s YOUR system. Design it however you like.

Most servers (if not all) can have more than one network card these days. And even if yours couldn’t – you can ask your network engineers to set up multiple virtual networks on top of the physical network and divide up the bandwidth between them.

Putting it together

The next step is to simply put the MarkAsFraud method on a different web service. This way, you can decide at deployment time which web service should be hosted over which network.

When your system is under load, you will then be guaranteed that even if there are a large number of low priority calls being invoked, they will not use up the network bandwidth reserved for your higher priority calls. You will likely still need to take care of processing and other IO concerns on your servers, but if worst comes to worst, you can partition your server farm as well.

While this may sound a bit CQRS-ish but it would be more accurate to say that CQRS is a more specific case of this pattern – that of partitioning the API according to business priority.

One of the interesting things about messaging is that we tend to forgo the traditional “service contracts” where many methods are put together on a single “service”. Instead, each message definition stands on its own and can be routed to any destination.

In summary

If you are still using WCF and web services, be aware that these apparently little things can have an impact on how your system behaves under load. Even if you do use MSMQ under WCF, the traditional service contract made of multiple methods will still govern your routing.

If you do go all the way with this pattern, you’ll see that each of your service contracts ends up with only one method on it. This might make you wonder what’s the point of the whole service contract thing in WCF – that would be a very good issue to resolve.

Remember the first rule of remote communication – Don’t.

One of the things which often throws people off when looking to identify their service boundaries is the UI design. Even those who know that the screen a user is looking at is the result of multiple services working together sometimes stumble when dealing with forms that users enter data into.

Let’s take for example a screen from the Marriott.com online reservation system (below). This screen collects information about the guest staying at the hotel (name, phone number, address, etc) and credit card information.

While we might have wanted to keep guest information in a separate service from the credit card information (which may very well be the corporate card of someone responsible for travel), the above screen would seem to indicate that the data would be collected together, validated together, and would also have to be processed together.

The traditional way

In standard layered architectures you would have all the data submitted by the user passed in a single call from a controller to some “service layer” (possibly running on a different machine), which would then persist that data in one transaction.

Even if some attempt was made to separate things out, there likely would be some “orchestration service” that received the full set of data and it would make calls to the other “services”, passing in the specific data that each “service” is responsible for.

I am putting quotes around the word “service” to indicate that I don’t consider these proper services in the SOA sense (as they lack the necessary autonomy) – they are more like functions or procedures, whether or not they’re invoked XML over HTTP is besides the point.

What to do?

Like so many other things, the solution is simple but a bit counter-intuitive as it doesn’t follow the way most web development is done, i.e. one submit button => one call to the server.

Let’s say the “Red” service is responsible for guest information and the “Blue” service is responsible for credit card data. In this case, each service would have its own javascript come down with the page and that script would register itself for a callback on the click of the submit button. Each service would take the data the user entered into its part of the page and independently make a call to “the” server (could be to 2 separate servers) where the data is persisted (potentially to 2 different databases).

This raises other questions, of course.

Now that the data submitted is being processed in 2 transactions rather than just one, we may need to figure out how to correlate the data. In this specific case, it’s not such a big deal as there is no direct relationship between the guest and the credit card – both need to be independently correlated to some reservation ID.

That reservation ID would likely have been “created” on a button click on a previous screen by some other service. The reason why I put the word “created” in quotes is that this could be as simple as having the client generate a new GUID and put that in a cookie (which would cause the reservation ID to end up being submitted along with subsequent requests). Another alternative would be to put the reservation ID in the session.

It’s quite possible that the reservation ID would only be persisted much later in the service that owns it when the user actually confirms the reservation on the website.

In any case, what we can see is that each of the commands of our respective services can now be processed independently of the others in an entirely asynchronous fashion thus vastly improving the autonomy of our services.

Some words on CQRS

This style of UI composition where services leverage javascript code running in the browser isn’t technically difficult in the slightest. The rest of the implementation of each service – having a controller that takes that data and passes it on for persistence can be quite simple.

I’d say even more strongly, most of the time you shouldn’t need to use any fancy-dancy messaging to get that data persisted – that is, unless you’re still stuck with the big relational database behind 23 firewalls type data tier. Embrace NoSQL databases for the simplicity and scalability they provide – don’t try to re-invent that using messaging, CQRS, persistent view models, event-sourcing, and other crap.

There are other very valid business reasons to embrace CQRS, but they have nothing to do with persistence.

Also notice, this is all happening within a service boundary / bounded context.

In closing

If you aren’t leveraging these types of composite UI techniques, it’s quite likely that your service boundaries aren’t quite right. Do be aware of the UI design and use it to inform your choices around boundaries, but be aware of certain programming “best practices” that may lead you astray with your architecture.

Also, if you’re planning on coming to my course in Toronto to learn more about these topics, just wanted to let you know that there’s one week left for the early-bird discount.

Finally, it’s good I have a birthday that comes around once a year to remind me that my time here isn’t unlimited and that I had better get off my rear and do something meaningful with the time I do have. If you get value from these posts, leave a comment or send me a tweet to let me know – it does wonders for my motivation.

Thanks a bunch.

Update – clarification post here.

I was on a call recently with the Advisory Board for the Microsoft Patterns & Practices (P&P) CQRS Journey project where they were showing the current state of their development. Towards the middle of the call, I mentioned that I found there to be too many concerns in one place and that I had expected there to be a division into multiple sub-domains/bounded contexts/business components (BCs). The answer was that they hadn’t gotten to the other areas yet and that’s why at that point in time there was only one BC.

The conversation got a bit derailed at that point, and I was asked how I would do it (though not quite as politely), ultimately leading to my tweeting this:

MS P&P CQRS project asked me to show how I would do the conf mgmt domain My Way. Anyone want to help me show them how to do it right?

— UdiDahan (@UdiDahan) March 21, 2012

I think I got over 50 people who wanted in on this, while some of them urged me to work with P&P rather than separately. I think I’ll do both, hopefully resulting in two implementations that can be compared – one based on Azure (done by P&P) and the other based on NServiceBus (done by my guys). Who do you think is more worried 😉

But first things first

The fundamental flaw that I see happening with many software projects (including the P&P CQRS effort) is that not enough time is spent to understand the underlying business objectives – the thinking behind the use cases / user stories. Developers assume behavior is “like” that of another/similar domain – when the difference in the details matter a lot. That often leads to software boundaries that aren’t properly aligned with those of the business.

The effects of this lack of alignment may be felt only much later in the project, when we get a requirement that just doesn’t fit the architecture we’ve set up. I’ve blogged about the symptoms of this problem about 2 years ago in my post Non-functional architectural woes.

We need to get into the nitty-gritty of our problem domain to find out what makes it special.

Not all e-commerce is equal

Anytime somebody is going to make a purchase online, developers immediately create some kind of “order” entity with a bunch of “order lines”, just like they read about in all the blog posts and books. Then, all sorts of other behavior are shoe-horned around those entities and… voila, a working system.

The domain of conferences is different – we don’t actually ship products when people register so payment concerns are very different. If our company is purchasing 5 tickets to the conference, the number of people (and which specific people) that eventually go to the conference may be very different than the people we had originally registered – there doesn’t tend to be that kind of volatility in traditional B2C retail (like selling books to people online).

It’s also quite likely that if a company is sending many people to a conference that they wouldn’t be paying by credit card – invoicing and payment may happen much later. That is no reason to block registration from completing.

Not all registration systems are equal

I understand how people can look at systems like TicketMaster and use that as a model for this system but, once again, the differences in the domain matter.

First of all, most people don’t purchase movie tickets weeks in advance – conference tickets do go on sale that far in advance. Second, if the movie you want to go to is sold out this week, no big deal, you’ll see it next week – conferences are more of a one-time/yearly deal. Third, you usually go to the movies with family/friends – if you can’t get tickets for everyone, you’ll go next week. When it comes to conferences, there is no “next week”, so whoever can go, does. Also, attendees going to a conference together are usually coworkers, not family – there are less qualms about leaving someone behind.

This is already leading us to a model where we should not view a group registration as a single success or failure affair. This will have an impact on the commands, events, and transactions that flow through our system.

In any case where people are reserving something far in advance, there is a high likelihood of cancellations. This is similar to the domain of hotels/hospitality where you can cancel your reservation up to N days before your arrival at no charge. This also tends to influence the payment structure – we’d rather not have to return people’s money as there can be per-transaction charges for that, instead delaying payment can make sense.

Similar to how hotels overbook by a certain amount (to offset cancellations), our conference might look at doing something similar. The difference is that in the case of a hotel, the guest will likely just book a room in a different hotel in the case the first hotel was fully booked. This probably won’t happen with a conference.

For that reason, we want to remember who wanted to come to our conference even when we thought we were full. You see, our best chance of filling a seat that opened up due to a cancellation is by a person who wanted to register before. What we need here is a waiting list – something that doesn’t make the same kind of sense for hotels or airlines (although airlines do use waiting lists, just that that is usually exposed to travel agents and not to travelers booking online).

First-come, first-served – fairness

The traditional developer thinking about systems is rooted in synchronous and sequential processes. In attempting to give a good user experience, developers want to give the user final confirmation as quickly as possible – whether that’s success or failure.

This results in a first-come, first-served user interaction model – whichever user registers in our conference management system first, the better the chance they’ll get what they want. That sounds like a pretty fair system, the only thing is that fairness was not a requirement.

In the real world, if people are standing in line for tickets, they’d get really upset if the tellers decided somewhat arbitrarily to serve people in the back of the line before those in the front. The great thing about online systems is that nobody can see the “virtual line” – the system can be as unfair as we like and there isn’t a real way for the users to know that this is happening.

Why be unfair?

While conferences, theaters, and airlines all want to have all seats filled, the difference between the ongoing models of airlines and theaters and the once-a-year model of the conference influence how sales are done. Some companies send a lot of employees to our conference so we want to give them preference in registration. This is area that we have the most leverage over – when it comes to the masses who arrive in ones and twos, there’s not very much we can do. It makes sense to bend over backwards for a large group, but not for a small one. A commitment from a large company tends to mean more than that from a small one.

If Boeing has already registered 70 people to your conference and now wants to send 5 more, are you really going to tell them “sorry, we’re fully booked”, or are you going to do everything in your power to keep them happy so that next year they’ll want to keep working with you? Wouldn’t it be nice if you could “unregister” some people to make room for the Beoing guys.

Now, you can’t necessarily do this up until the last minute, but potentially 2 weeks (or whatever) before the event could be reasonable, leaving people the ability to cancel flights and hotels without charges (assuming we tell them during registration that they should buy refundable plane tickets).

The easiest way to “unregister” someone is to not tell them that their registration was confirmed. In short, 2 weeks before the start of the event we finalize all registrations deciding (based on our internal priority) who gets in and who doesn’t. We may have logic that decides to immediately finalize registration from Boeing (and other select customers) without waiting until 2 weeks before the event.

Just don’t look TOO unfair

Appearance is everything. Perception matters. You don’t want to get a reputation for being unfair.

So when we open registration, we can allow the first N people to bypass our waiting list and get accepted right away (payment still needing to be handled later). At that point, you can start moving new registrations through the waiting list.

The thing is, nobody knows that you aren’t actually full at that point 🙂

Influences on architecture

I hope you’re getting the impression that this collection of scenarios is going to have a big impact on the design. It indicates to us which parts of the business need to be 100% consistent with each other and which parts can be eventually consistent – ultimately defining where one bounded context stops and another begins. This has a direct impact on the events that we’d end up with – who would publish what, and how many others would subscribe to it.

I know some people will look at the above scenarios and say “but what if the requirements were different?”. The thing is that not all requirements are created equal. In working with our business stakeholders, we need to identify which elements are stable and which are potentially volatile and, yes, that’ll be different in each project. We want to align the main boundaries of our software with the stable business elements.

And don’t even try to create a system so flexible that it could handle any new requirement without any architectural changes – down that path lies madness. User-defined custom fields used in user-defined custom workflows, all of it appearing in reports with sorting, filtering, and grouping. You might as well give your users Visual Studio.

Back to P&P

I don’t know if P&P will adopt this set of requirements for their CQRS Journey. The thing here is that we can see the collaborative nature of the domain quite clearly – multiple actors working in parallel where the decisions of one affect the outcomes of another.

The requirements that I’ve seen being handled in the CQRS Journey so far don’t seem complicated enough to justify anything more than a 2-tier architecture – it’s feeling somewhat over-engineered right now. I know that people in the community see other benefits to CQRS but I’ll have to put up a separate blog post describing why there are other better solutions than CQRS most of the time.

Anyway, I’m willing to see how things progress and tweak these requirements (up to a point) so that both the NServiceBus solution and the Azure solution are addressing the same problem.

In closing

Occasionally I hear people still raising the agile mantra against Big Design/Requirements Up Front. The thing is that Agile Manifesto never said to intentionally bury your head in the sand with regards to the purpose of the system. It was a push-back against spending months in analysis without anything but documents coming out, but the goal was to reach a middle ground. Nobody ever said “no design up-front” or “no requirements up front”.

I’m going to try to work with both P&P and the alumni of my Advanced Distributed Systems Design course to come up with simplest possible solution that addresses the requirements (functional and non-functional).

Hope you’ll find this journey interesting.

Update – clarification post here.

Abuse #1

“I’m using CQRS because I need to scale.”

While CQRS may be more scalable than other more traditional architectures, the use of asynchronous communication often complicates the user interaction model causing users to not see the changes they made to data in the UI until later. Trying to compensate for this (by writing even more code) digs one deeper into the complexity hole.

When I point to non-collaborative subdomains and state “You don’t need CQRS for that”, the reason is that in these areas you don’t tend to have much read/write contention. While multiple users/actors may be working in parallel, they don’t touch the same set of data (or do so only very rarely).

In these environments, all you need is a scalable data storage technology – something designed to scale-out (unlike most relational databases). This can take the form of NOSQL databases like HBase and Cassandra. Often all you need is the UI to query that directly and show the results, and the same goes for persisting the data back – possibly with some basic validation and calculation code on the side.

No commands, events, DTOs, publish/subscribe, domain model, etc.

As Ayende says – JFHCI, just f-ing hard code it.

You’d be surprised how much of your data this approach can apply to.

With the time you save on all the less important stuff, you’ll have more time to apply CQRS the right way for the high-value/high-complexity parts of your system.

***

Just a final note, as registration for my course in New York is coming to a close in 2 weeks, I wanted to let you all know that the price for the course will be going up this April, after the course in Sydney. The reason for this is that the courses I run myself (at the current rate) have been cannibalizing attendees from the partner companies I do the course with.

I’ll be providing significant discounts to independent consultants (and others paying their own way) to try to keep things fair. Hope to see you there.

Go to the registration page.

Hard to believe, isn’t it?

Although both myself and Greg have been saying (quite publicly) for a long time now that we’re in agreement in about 99% of the DDD/CQRS content we talk about, it turns out the terminology we use has made it very difficult for everybody else to see that.

Anyway, on a recent call with Greg and the Microsoft Patterns & Practices team working on the CQRS guidance, I think we finally ironed out the terminological differences.

First of all, both of us clearly stated that CQRS is not meant to be the top-level architecture of a system.

The use of Bounded Contexts from Domain Driven Design is a good way to *start* handling that top-level.

The area of some contention was how big a Bounded Context should be. After going back and forth a bit, Greg brought the concept of Business Component into the conversation, and that really cleared things up all around. I was quite pleased as I’ve been going on and on about these business components for years (I think 2006 was one of my earlier posts on the topic, though the mp3 has disappeared since then).

Anyway, here’s the meat:

A given Bounded Context should be divided into Business Components, where these Business Components have full UI through DB code, and are ultimately put together in composite UI’s and other physical pipelines to fulfill the system’s functionality.

A Business Component can exist in only one Bounded Context.

CQRS, if it is to be used at all, should be used within a Business Component.

There you have it – terminological agreement in addition to the philosophical agreement that was always there.

You can find the history of my posts mentioning Business Components here.

Last month both myself and Eric Evans spoke at a conference run by the International Association of Software Architects (IASA) in Madrid. Eric talked about DDD and I talked about CQRS. While the talks were recorded, I don’t think they’ve come online yet.

At the end of the conference, we were interviewed by the local .NET magazine dNM and that video is now available here. We covered the background on things like DDD, CQRS, and the Cloud. I don’t think that either of us said anything earth-shattering but if you have half an hour, take a look:

… but differently than the way most people have been using it.

I think I’ve just about drove everybody crazy now with my apparent zigzagging on CQRS.

Some people heard about CQRS first from one of my presentations and got all excited about it. Then I did some blogging which further drove people to CQRS (as did Greg Young and some others). As CQRS was just about to hit its stride with the Early Adopters, I started pushing a more balanced view – CQRS not as an answer, but as one of many questions. More recently I’ve pushed more strongly back against CQRS saying that it should be used rarely.

So what’s the missing piece?

If you’re in the Domain-Driven Design camp (as many doing CQRS are), then it’s Bounded Contexts.

If you’re in the Event-Driven SOA camp (a much smaller camp to be sure), then it’s Services.

The problem is the naming, because the DDD guys have their kinds of services which do not fit the definition for Service of the Event-Driven SOA approach.

Let me propose the term Autonomous Business Component for the purposes of this blog post to describe that thing which is both a DDD Bounded Context (have the shared BC part of the acronym) and an SOA Autonomous Services. Resulting in the nice short form: ABC (and everyone knows you need to have a good acronym if you want something to catch on).

What does this have to do with CQRS?

Nothing just yet. Well, at least, nothing directly to do with CQRS.

Although some proponents of CQRS have stated that it can and should be used as the top-most architectural pattern, both myself and Greg Young (arguably the first two to talk about it and the two who ultimately collaborated on naming it – and now Google knows we didn’t means “cars”) always recommended it as a pattern to be used one level down.

Although Greg and I have had many long discussions on the topic and do agree very much about what the overall structure should look like, I’ll try to avoid putting words in his mouth from this point on.

Before talking more about ABCs, let’s discuss the principle upon which they rest: The Single Responsibility Principle (SRP).

What does SRP have to with CQRS?

Many developers are familiar with SRP and have seen good results from using it. What we’re going to do is take this principle to the next level.

In Object Orientation (OO), data is encapsulated in an object. A good object does not expose its data to other objects to do with as they wish. Rather, it exposes methods that other objects can invoke, and those methods operate on the internal data.

SRP would guide us to not have the same data exist in two objects. For example, if we saw the customer’s first name as an internal data member of two objects, we’d be right to question that kind of duplication and move to refactor it away. However, when we see two systems doing the exact same thing – somehow that gets excused.

“Of course we need to be able to see the customer’s first name in the front-end website as well as in the back-end fulfillment system. How could we NOT have the customer’s first name in both those code-bases?”

And there’s the catch.

Who said that a system should be a single code-base?

But what about integration?

Although many times we do need to integrate existing systems together, sometimes we have the ability to change those systems. More importantly, when going to create a new solution, we can avoid getting ourselves into the problems that integration tries to solve.

Integrating with a system that cannot be changed can be done also by composing multiple ABCs, but that’s a topic for another post.

It is better to think of integration as a necessary evil – kind of like regular expressions and multi-threading; things to be avoided unless absolutely necessary.

“If you have a problem that you decide to use a regular expression to solve, you now have 2 problems.” Or so the saying goes. With multi-threading, you have a non-deterministic number of problems to solve.

If you thought you had duplicate responsibilities with 2 systems operating on the same data, how will introducing a 3rd code base (also known as “integration”) help? Remember that Single Responsibility Principle – our goal is to get it down to one.

OK, so how do ABCs do that?

In order for us to get back into alignment with SRP, that would require us to have responsibility for a single piece of data exist in one code base. Note that SRP makes no statements about how many physical places a given code base can be deployed to. Nor does it state that only a single technology can be in play – code that emits HTML can be packaged at design time together with rich-client code in the same solution.

If an ABC is responsible for a piece of data, it is responsible for it everywhere, and forever. No other ABC should see that data. That data should not travel between ABCs via remote procedure call (RPC) or via publish/subscribe. It is the ultimate level of encapsulation – SRP applied at the highest level of granularity.

This results in systems which are the result of deploying the components of multiple ABCs to the same physical place. The ABC which owns the customer name would have the necessary web code to render it in the e-commerce front-end and in the shipping back-end for printing on labels. This would mean that practically every screen in any UI is a composite of widgets owned by their respective ABCs.

This is ultimately what keeps the complexity of each ABC’s code base to a minimum.

But why not just use CQRS as the top-level pattern? ABCs are weird.

Imagine trying to create a single denormalized view model for the entire Amazon.com product page – product name, price, inventory, editorial review, customer comments, other products that customers viewed, other products that customers bought, etc.

Pretty complex, right?

How much duplication would you have for the page shown after you add an item to a cart? Once again, you need to show other products that customers bought, their names, images, prices, and inventory.

And then on the home page – items you might be interested in, names, images, prices.

And that’s only in the front-end system.

It’s not just the duplication, but how complex the code is for each one.

Instead of the duplication that top-level CQRS would bring you, consider an ABC responsible for products names and images that has just about the same view model composed on each of the above screens. The same with another ABC responsible for price.

You may be thinking that this would result in more queries to get the data to show on a page, and you’d be right. But it isn’t necessarily a classical N+1 Select problem, as the queries are bounded to the number of ABCs. Secondly, consider the ability to have well-tuned caching at the granularity of an ABC – something that would be much more difficult when dealing with everything as a single monolithic view model. In short, not only will it not be a performance problem, often it will actually improve performance.

OK – that explains “everywhere”, what about “forever”?

Forever is where things get interesting – or more accurately, when they get interesting.

Let’s talk about things like invoices.

One of the requirements in this area is that immutability. If the customer’s name was Jane Smith when they made their purchase, it doesn’t matter that they’ve since changed their name to Jane Jones, the invoice should still show Jane Smith.

Often developers push these types of requirements on the data warehouse guys – that’s where history gets handled. The only thing is that if your ABC owns the customer’s name, then no other code base can deal with it. If it’s your data, you have to handle all historical representations of it.

On the one hand, this would seem to kill the data warehouse. On the other hand, it means that the principles of data warehouses are now core to every code-base.

This means you don’t ever delete data (see my previous blog post on the subject), and you definitely don’t overwrite it with an update – even if you think you’re in a simple CRUD domain. The only case where you can get away with traditional CRUD is if we’re talking about private data – data that is only ever acted on by a single actor.

This sounds like the collaboration you talk about with CQRS

It’s similar in principle but different in practice.

In a collaborative domain, an inherent property of the domain is that multiple actors operate in parallel on the same set of data. A reservation system for concerts would be a good example of a collaborative domain – everyone wants the “good seats” (although it might be better call that competitive rather than collaborative, it is effectively the same principle).

A customer’s name would not fall under that category. It isn’t an inherent property of the domain for multiple actors to operate on that data. While there can be multiple readers, one can easily enforce a single writer without any adverse effects. Doing that with a reservation system would cause the online system to behave as if users were lining up in front of a box office – not a desirable outcome.

Private data would be something like a user’s shopping cart. Until they make a purchase, that data doesn’t need to be visible anywhere. Here you could theoretically do simple CRUD – that is, until the business realizes that there’s extremely valuable information to be extracted from the historical record of things people do with their carts.

I think you’re ready to make your point, so just make it already

OK – so we now realize that Update and Delete don’t exist in their traditional form. Delete is really just a kind of update, and update is effectively an “upsert” – a combination of update and insert to retain history. This can be done by having ValidFrom and ValidTo columns for our data.

In which case, Create is really just a special case of Upsert, which looks like this:

UPDATE Something SET ValidTo = NOW() WHERE Id=@Id AND ValidTo = NULL; INSERT INTO Something SET { regular values }, Id=@Id, ValidTo = NULL;

And then we’d have 2 forms of Read – reading the current state (ValidTo = NULL), and reading history (ValidFrom <= Instant AND (ValidTo >= Instant OR ValidTo = NULL))

Here we don’t need fancy N-Tier architectures, data transfer objects, service layers, or domain models. A simple 2-Tier approach could probably suffice. We don’t need a task-based UI, events, denormalized view models, or any of that CQRS stuff. This was at the crux of my previous anti-CQRS post.

The only thing is that this is exactly CQRS.

Say what?

Have we not effectively separated the responsibility of commands/upserts and queries/reads?

As Greg Young has said before, “the creation of 2 objects where there previously was one”.

Effectively 2 paths through our ABC.

CQRS.

Let me give you a second to gather your thoughts.

*

You see, CQRS is an approach, a mind-set – not a cookie cutter solution. Frameworks that guide you to applying CQRS exactly the same way everywhere are taking you in the wrong direction. The fact is that you couldn’t possibly know what your Aggregate Roots were before you figured out how to break your system down into ABCs. Attempting to create commands and events for everything will make you overcomplicate your solution.

So the built-in history of this model is event-sourcing?

Well, it’s not event-sourcing in the sense that we don’t necessarily have events. It achieves many of the benefits of event-sourcing by giving us the full history of what happened.

On the whole issue of replaying events to fix bugs – that’s a bit problematic, logically, unless we have a closed system. A closed system is one that doesn’t interact with anything else – no other systems, no users, nothing. As such, closed systems aren’t that common.

In an open system, one with users, let’s say there was a bug. This bug could have caused the wrong data to be written and/or shown to users. As such, users could have submitted subsequent commands based on that erroneous data that they would not have submitted otherwise. There’s no way for us to know.

The problem with replaying events when we fix the bug is that we’re in essence rewriting history – making it as if the user didn’t see the wrong data. The only problem is that we can’t know which events not to replay – we can’t automatically come up with the right events that should have come afterwards. We could try to sit together with our users and have them try to revise history manually, but our organization often isn’t in a bubble. Our users interacted with customers and suppliers. It isn’t feasible to try to undo the real-world impacts of this situation.

Why didn’t you just tell us this from the very beginning?

I did, you just weren’t listening.

You wanted a cookie cutter, and until you tried CQRS out as cookie cutter (and saw it create a bunch of complexity) you wouldn’t listen to anything else.

As developers, we’re trained to solve problems – the faster the better. Unfortunately, this causes us to be blind to things that don’t immediately present themselves as solutions.

When applying CQRS with ABCs, the solutions you end up with are very simple, but the process of getting there is quite hard and takes practice. Finding the boundaries of ABCs such that data isn’t duplicated between them and that data doesn’t travel between them either via RPC or publish/subscribe – it may feel impossible the first several times you try. Keep at it – it is almost always possible.

We haven’t touched on the whole saga/aggregate-root thing yet, but that isn’t as important until you can successfully apply the principles described here.

Also, this post has already gotten long enough, so it looks like now would be a good time to stop.

Until next time…