Why you should be using CQRS almost everywhere…
Sunday, October 2nd, 2011.
… but differently than the way most people have been using it.
I think I’ve just about drove everybody crazy now with my apparent zigzagging on CQRS.
Some people heard about CQRS first from one of my presentations and got all excited about it. Then I did some blogging which further drove people to CQRS (as did Greg Young and some others). As CQRS was just about to hit its stride with the Early Adopters, I started pushing a more balanced view – CQRS not as an answer, but as one of many questions. More recently I’ve pushed more strongly back against CQRS saying that it should be used rarely.
So what’s the missing piece?
If you’re in the Domain-Driven Design camp (as many doing CQRS are), then it’s Bounded Contexts.
If you’re in the Event-Driven SOA camp (a much smaller camp to be sure), then it’s Services.
The problem is the naming, because the DDD guys have their kinds of services which do not fit the definition for Service of the Event-Driven SOA approach.
Let me propose the term Autonomous Business Component for the purposes of this blog post to describe that thing which is both a DDD Bounded Context (have the shared BC part of the acronym) and an SOA Autonomous Services. Resulting in the nice short form: ABC (and everyone knows you need to have a good acronym if you want something to catch on).
What does this have to do with CQRS?
Nothing just yet. Well, at least, nothing directly to do with CQRS.
Although some proponents of CQRS have stated that it can and should be used as the top-most architectural pattern, both myself and Greg Young (arguably the first two to talk about it and the two who ultimately collaborated on naming it – and now Google knows we didn’t means “cars”) always recommended it as a pattern to be used one level down.
Although Greg and I have had many long discussions on the topic and do agree very much about what the overall structure should look like, I’ll try to avoid putting words in his mouth from this point on.
Before talking more about ABCs, let’s discuss the principle upon which they rest: The Single Responsibility Principle (SRP).
What does SRP have to with CQRS?
Many developers are familiar with SRP and have seen good results from using it. What we’re going to do is take this principle to the next level.
In Object Orientation (OO), data is encapsulated in an object. A good object does not expose its data to other objects to do with as they wish. Rather, it exposes methods that other objects can invoke, and those methods operate on the internal data.
SRP would guide us to not have the same data exist in two objects. For example, if we saw the customer’s first name as an internal data member of two objects, we’d be right to question that kind of duplication and move to refactor it away. However, when we see two systems doing the exact same thing – somehow that gets excused.
“Of course we need to be able to see the customer’s first name in the front-end website as well as in the back-end fulfillment system. How could we NOT have the customer’s first name in both those code-bases?”
And there’s the catch.
Who said that a system should be a single code-base?
But what about integration?
Although many times we do need to integrate existing systems together, sometimes we have the ability to change those systems. More importantly, when going to create a new solution, we can avoid getting ourselves into the problems that integration tries to solve.
Integrating with a system that cannot be changed can be done also by composing multiple ABCs, but that’s a topic for another post.
It is better to think of integration as a necessary evil – kind of like regular expressions and multi-threading; things to be avoided unless absolutely necessary.
“If you have a problem that you decide to use a regular expression to solve, you now have 2 problems.” Or so the saying goes. With multi-threading, you have a non-deterministic number of problems to solve.
If you thought you had duplicate responsibilities with 2 systems operating on the same data, how will introducing a 3rd code base (also known as “integration”) help? Remember that Single Responsibility Principle – our goal is to get it down to one.
OK, so how do ABCs do that?
In order for us to get back into alignment with SRP, that would require us to have responsibility for a single piece of data exist in one code base. Note that SRP makes no statements about how many physical places a given code base can be deployed to. Nor does it state that only a single technology can be in play – code that emits HTML can be packaged at design time together with rich-client code in the same solution.
If an ABC is responsible for a piece of data, it is responsible for it everywhere, and forever. No other ABC should see that data. That data should not travel between ABCs via remote procedure call (RPC) or via publish/subscribe. It is the ultimate level of encapsulation – SRP applied at the highest level of granularity.
This results in systems which are the result of deploying the components of multiple ABCs to the same physical place. The ABC which owns the customer name would have the necessary web code to render it in the e-commerce front-end and in the shipping back-end for printing on labels. This would mean that practically every screen in any UI is a composite of widgets owned by their respective ABCs.
This is ultimately what keeps the complexity of each ABC’s code base to a minimum.
But why not just use CQRS as the top-level pattern? ABCs are weird.
Imagine trying to create a single denormalized view model for the entire Amazon.com product page – product name, price, inventory, editorial review, customer comments, other products that customers viewed, other products that customers bought, etc.
Pretty complex, right?
How much duplication would you have for the page shown after you add an item to a cart? Once again, you need to show other products that customers bought, their names, images, prices, and inventory.
And then on the home page – items you might be interested in, names, images, prices.
And that’s only in the front-end system.
It’s not just the duplication, but how complex the code is for each one.
Instead of the duplication that top-level CQRS would bring you, consider an ABC responsible for products names and images that has just about the same view model composed on each of the above screens. The same with another ABC responsible for price.
You may be thinking that this would result in more queries to get the data to show on a page, and you’d be right. But it isn’t necessarily a classical N+1 Select problem, as the queries are bounded to the number of ABCs. Secondly, consider the ability to have well-tuned caching at the granularity of an ABC – something that would be much more difficult when dealing with everything as a single monolithic view model. In short, not only will it not be a performance problem, often it will actually improve performance.
OK – that explains “everywhere”, what about “forever”?
Forever is where things get interesting – or more accurately, when they get interesting.
Let’s talk about things like invoices.
One of the requirements in this area is that immutability. If the customer’s name was Jane Smith when they made their purchase, it doesn’t matter that they’ve since changed their name to Jane Jones, the invoice should still show Jane Smith.
Often developers push these types of requirements on the data warehouse guys – that’s where history gets handled. The only thing is that if your ABC owns the customer’s name, then no other code base can deal with it. If it’s your data, you have to handle all historical representations of it.
On the one hand, this would seem to kill the data warehouse. On the other hand, it means that the principles of data warehouses are now core to every code-base.
This means you don’t ever delete data (see my previous blog post on the subject), and you definitely don’t overwrite it with an update – even if you think you’re in a simple CRUD domain. The only case where you can get away with traditional CRUD is if we’re talking about private data – data that is only ever acted on by a single actor.
This sounds like the collaboration you talk about with CQRS
It’s similar in principle but different in practice.
In a collaborative domain, an inherent property of the domain is that multiple actors operate in parallel on the same set of data. A reservation system for concerts would be a good example of a collaborative domain – everyone wants the “good seats” (although it might be better call that competitive rather than collaborative, it is effectively the same principle).
A customer’s name would not fall under that category. It isn’t an inherent property of the domain for multiple actors to operate on that data. While there can be multiple readers, one can easily enforce a single writer without any adverse effects. Doing that with a reservation system would cause the online system to behave as if users were lining up in front of a box office – not a desirable outcome.
Private data would be something like a user’s shopping cart. Until they make a purchase, that data doesn’t need to be visible anywhere. Here you could theoretically do simple CRUD – that is, until the business realizes that there’s extremely valuable information to be extracted from the historical record of things people do with their carts.
I think you’re ready to make your point, so just make it already
OK – so we now realize that Update and Delete don’t exist in their traditional form. Delete is really just a kind of update, and update is effectively an “upsert” – a combination of update and insert to retain history. This can be done by having ValidFrom and ValidTo columns for our data.
In which case, Create is really just a special case of Upsert, which looks like this:
UPDATE Something SET ValidTo = NOW() WHERE Id=@Id AND ValidTo = NULL; INSERT INTO Something SET { regular values }, Id=@Id, ValidTo = NULL;
And then we’d have 2 forms of Read – reading the current state (ValidTo = NULL), and reading history (ValidFrom <= Instant AND (ValidTo >= Instant OR ValidTo = NULL))
Here we don’t need fancy N-Tier architectures, data transfer objects, service layers, or domain models. A simple 2-Tier approach could probably suffice. We don’t need a task-based UI, events, denormalized view models, or any of that CQRS stuff. This was at the crux of my previous anti-CQRS post.
The only thing is that this is exactly CQRS.
Say what?
Have we not effectively separated the responsibility of commands/upserts and queries/reads?
As Greg Young has said before, “the creation of 2 objects where there previously was one”.
Effectively 2 paths through our ABC.
CQRS.
Let me give you a second to gather your thoughts.
*
You see, CQRS is an approach, a mind-set – not a cookie cutter solution. Frameworks that guide you to applying CQRS exactly the same way everywhere are taking you in the wrong direction. The fact is that you couldn’t possibly know what your Aggregate Roots were before you figured out how to break your system down into ABCs. Attempting to create commands and events for everything will make you overcomplicate your solution.
So the built-in history of this model is event-sourcing?
Well, it’s not event-sourcing in the sense that we don’t necessarily have events. It achieves many of the benefits of event-sourcing by giving us the full history of what happened.
On the whole issue of replaying events to fix bugs – that’s a bit problematic, logically, unless we have a closed system. A closed system is one that doesn’t interact with anything else – no other systems, no users, nothing. As such, closed systems aren’t that common.
In an open system, one with users, let’s say there was a bug. This bug could have caused the wrong data to be written and/or shown to users. As such, users could have submitted subsequent commands based on that erroneous data that they would not have submitted otherwise. There’s no way for us to know.
The problem with replaying events when we fix the bug is that we’re in essence rewriting history – making it as if the user didn’t see the wrong data. The only problem is that we can’t know which events not to replay – we can’t automatically come up with the right events that should have come afterwards. We could try to sit together with our users and have them try to revise history manually, but our organization often isn’t in a bubble. Our users interacted with customers and suppliers. It isn’t feasible to try to undo the real-world impacts of this situation.
Why didn’t you just tell us this from the very beginning?
I did, you just weren’t listening.
You wanted a cookie cutter, and until you tried CQRS out as cookie cutter (and saw it create a bunch of complexity) you wouldn’t listen to anything else.
As developers, we’re trained to solve problems – the faster the better. Unfortunately, this causes us to be blind to things that don’t immediately present themselves as solutions.
When applying CQRS with ABCs, the solutions you end up with are very simple, but the process of getting there is quite hard and takes practice. Finding the boundaries of ABCs such that data isn’t duplicated between them and that data doesn’t travel between them either via RPC or publish/subscribe – it may feel impossible the first several times you try. Keep at it – it is almost always possible.
We haven’t touched on the whole saga/aggregate-root thing yet, but that isn’t as important until you can successfully apply the principles described here.
Also, this post has already gotten long enough, so it looks like now would be a good time to stop.
Until next time…
|
If you liked this article, you might also like articles in these categories: 
If you've got a minute, you might enjoy taking a look at some of my best articles.I've gone through the hundreds of articles I've written over the past 6 years and put together a list of the best ones as ranked by my 5000+ readers. You won't be disappointed.  If you'd like to get new articles sent to you when they're published, it's easy and free.Subscribe right here.  Follow me on Twitter @UdiDahan.
Something on your mind? Got a question? I'd be thrilled to hear it. Leave a comment below or email me, whatever works for you. 57 CommentsYour comment... |
May 7th, 2014 at 11:08 am
Gilligan,
In cases where you have more/many systems for which you don’t have the code or the ability to have the code changed, you need to be even more careful about responsibilities – especially those systems which provide all sorts of configuration/customization/extension abilities. That very quickly leads to all sorts of duplicated responsibility.
May 7th, 2014 at 9:30 pm
I understand, but how would you be able to manage those responsibilities without duplicating/caching that non-owned data if they don’t have historical representations?
May 10th, 2014 at 7:54 pm
It is OK to replicate data in multiple physical locations as long as you’re still in the scope of the same logical responsibility.
See: http://www.udidahan.com/2012/08/28/data-duplication-and-replication/
June 4th, 2014 at 7:23 pm
How do you do the whole status date thing if you have a list of items and each item has a different status date? A query per item I would imagine would be very poor performance, and would be forced to rely on some huge caching. e.g. I get a list of Orders and each order and I need to ger the Supplier Name at the time each Order was placed
March 5th, 2015 at 4:22 pm
An ABC is responsible for its historical data even if it is immutable, right? So, if that ABC handles a huge amount of historical data is acceptable to it to store that data into an aggregated format to increase performance? The ABC is responsible for all type of representations of its historical data even if that representation is used only to make a component more performatic?
March 7th, 2015 at 7:33 pm
Neylor,
Right on all accounts.
March 7th, 2015 at 7:35 pm
Gilligan,
Usually when doing a query, you know the context – data as it is right now vs. historical. Internally, the ABC may keep the “data as it is right now” either as a materialized view (in Oracle language) or a simple copy of the data to make those queries run quickly.