Well, I’m on my way back from Germany now after two intellectually-filled days at the Prio Conference in Baden-Baden. I finally got to meet Ralf Westphal with whom I’ve been emailing back and forth for over a year now. I saw Christian Weyer and Dominick Baier again after meeting them at TechEd Developers Barcelona. You can imagine that the conversations centered around large-scale distributed systems design.

On my first night there, after a delicious speaker dinner and several drinks Ralf and I waned technical. After spinning through some pub/sub, Ralf challenged me that he could implement all of the concepts I used between services and within them, client-side included, with holons (read Ralf’s blog for some background on this).

After debating back and forth a bit, I realized that almost all of the components in the solutions I design have a Command / Notify style interface. My persistent domain objects followed the Domain-Driven Design practices espoused in Eric Evans book (by the same name), and ended up exposing just such an interface.

High-level communications between services followed the same principles. Services received command messages and notified external services about state changes by publishing notification messages.

Even the repository in which objects were stored on the client side exposed the same style of interface. The methods “Save” and “Delete” caused the repository to raise a “RepositoryChanged” event. The only part of the repository that didn’t follow this pattern was requesting an entity based on its Id.

While it may not be the unified theory of everything, I was beginning to believe that this could be used as a general design principle – effective for teaching new architects how to get started. By combining this with my first principle of design (formerly known as Udi’s first rule of design) – interaction between classes must be through an interface, with each class and interface in a separate package – I think that we have a firm, yet simple foundation for architects to build on.

Just as an aside, I’ve changed the word “rule” to “principle” because there is a general case where it should not be used. When you have a class that fulfills a role like Supervising Controller, or Message Handler and it needs to interact with classes fulfilling the role of Domain Object – either server-side or client-side, the “rule” should not be used as it will cause unnecessary complexity with no benefit.

Finally, Ralf left me with an interesting idea to mull over. Given that both in-process and inter-process interactions follow similar patterns when loosely-coupled, could we create a design which could be distributed in multiple ways differing only in the system-level quality attributes achieved. My current thinking is that were you to do such a design when thinking about distribution scenarios, you could be able to co-locate all elements on a single machine, or even a single process, and have it work. I’m not sure that if you did such a design when thinking about single-process deployment the result could scale out as well as the first design. I’ve done the former and been successful, but haven’t tried the latter yet.

Anyway, at that point it was already late and I was tired after spending most of the day changing planes, and trains, and even cabs, so I called it a night – dreaming about component, service, and holon societies.

So, last night I was at the MVP dinner in TechEd and everything was nice. We had a nice meal, conversation was nice, weather was… nice. And then the volume started to rise, slowly at first, so as you don’t quite notice it. After a bit, you kind of stop talking and look around. And then I hear it…

<WWF announcer voice>
Are you ready… to RUMBLE !?!?
</WWF announcer voice>

It was Datasets vs. O/R mapping, a slight twist on the infamous datasets vs. custom objects debate, all over again. They pulled me in, kicking and screaming, I swear, I really do. The lines were drawn, maintainability, performance, all the things that architects like to philosophize about in terms of other people’s work.

Anyway, I won’t give you the play-by-play ‘cause we were there almost all night. I’ll just cut to the chase.

First things first – any comparison of solutions without the context of a problem leads nowhere, fast, and stays there. So the first question I asked (when I got the chance to speak) was “are we talking about querying/reporting here?” and the answer was something like “well, yeah, but a lot of other things too”. So my suggestion was that we discuss the solutions in terms of two contexts – querying/reporting and OLTP.

What I mean by OLTP is the data-updating kind of work that you do on certain items. Examples of this include “insert order”, “change customer address”, and “discount product”. Querying/reporting doesn’t change data, and often involves dealing with large sets of data pulled from different kinds of entities (in ERD terms).

Luckily, my suggestion to deal with them separately was accepted. Secondly, I proposed that an object model (specifically implementing the Domain Model pattern) designed for OLTP would perform poorly when used for querying/reporting – simply because it wasn’t designed for it. The structure of a domain model is such that it makes it possible to define / implement business rules in one place. That’s possible, not easy.

Well, the dataset people weren’t going to just hand me the OLTP side of the equation without a fight, so they mentioned how easy it was to just “AcceptChanges”, and that my way was much more complex. My rebuttal came in the form of a question (are you seeing a pattern here?): Do you just swallow DbConcurrencyExceptions are do you throw all the user’s changes away when it happens? I didn’t quite make out the answer since there was a lot of mumbling going on, but I’m pretty sure they had one. I mean, you can’t develop multi-user systems using datasets without running into this situation.

The example that clinched OLTP was this. Two users perform a change to the same entity at the same time – one updates the customer’s marital status, the other changes their address. At the business level, there is no concurrency problem here. Both changes should go through. When using datasets, and those changes are bundled up with a bunch of other changes, and the whole snapshot is sent together from each user, you get a DbConcurrencyException. Like I said, I’m sure there’s a solution to it, I just haven’t heard it yet.

Now, here’s where things get interesting. I didn’t say that using a domain model automatically solves this problem. Rather, I described how each client could send a specific message, one a ChangeMaritalStatusMessage, the other a ChangeAddressMessage, to the server – in essence, giving the server the context in which each bit of data is relevant. The server could just open a transaction, get the customer object based on its Id, call a method on the customer (ChangeMaritalStatus or ChangeAddress), and commit the transaction. If two of these messages got to the server at the same time, the transactions would just cause them to be performed serially, and both transactions would succeed. The important part here is not losing the context of the changes.

When we talked about querying/reporting, things seemed quite a bit clearer. Datasets, or rather datatables seemed like a fine solution – most 3rd party controls support them out of the box. One guy mentioned that datasets performed poorly for large sets of data and that by designing custom entities for the result set, he could improve performance and memory utilization by, like, 70%. To tell you the truth, I think that if you need the performance, do it, if not, just use datasets. There isn’t much of an issue of correctness.

Just as an ending comment, in response to something someone said about scalability, I asked if they were reporting against the live OLTP data. The response was “yes”. Well, there’s a database scalability problem if I ever saw one. OLTP works most correctly when employing transactions that have an isolation level of serializable. The problem with them is that they lock up the whole table, or get blocked when a table scan is going on. Querying often results in a table scan. You can see the problem. Anyway, a common solution to this problem is to just reduce the isolation level, a quick fix that improves performance almost immediately. You take one hit in that your reports may be showing incorrect data, especially if they do aggregate type work. You might take another hit if your OLTP transactions need to do aggregate type work themselves. That second hit is pretty much unacceptable. A different solution is to accept the fact that the heaviest querying can usually show data that isn’t up to date up to the second.

In such a solution, you would have another database for reporting. It wouldn’t be just a replica of the OLTP database, but rather a lot more denormalized – which is a really not nice way of saying designed for reporting. You could then move the data from your OLTP database to the reporting database in some way (more to come on this topic) and you increase the scalability of your database. Just to define that a bit better – your OLTP database will be able to handle more transactions per unit of time, and reports will run faster, meaning that you will both improve their latency and the number of queries that can be handled per unit of time.

Anyway, I was pretty tired after all that, but if I had to sum it up I’d say something like this: before debating solutions, define the problem, you get a lot more insight into the solutions and you get it faster. That’s just win-win all around.

I received the following question from Houman and I thought the both the question and answer deserved some more public discussion:

“Hello Udi,

Regards,
Houman”

And my response:

—
Consistency is a domain level concept – for instance, when you order a vacation including hotel, flight, and car rental, if one of those orders falls through, you don’t want to “rollback” on the others. From these domain level definitions, you can often run a sanity check on your service boundaries. Areas of consistency should fall within the boundaries of a single service. I don’t see myself, or anyone else for that matter, trading off on this – it is fundamental to the correctness of your system.

Availability is, or at least should be, orthogonal. You should be able to be consistent AND available at the same time. Service availability is not much different from “regular” availability that we’ve been doing all along. Of course, if we’ve divided up our services like I said before, than when one service becomes unavailable for some reason, the availability of other services should not be affected. This is definitely a part of service autonomy. If you have a poor division of responsibility between services, you can see it manifesting itself when the unavailability of one service causes other services, even though they may still run, to be incapable of providing service.

WCF is simply a communication mechanism. By understanding the above concepts, you will know which part of WCF to employ and when in your specific application. Workflow Foundation (WF) doesn’t really relate to any of the above.
—

Via the “What to Fix” blog, Signs you have too much process. I totally connected with the last bullet:

“You’ve reached CMMI level 7, in which you document reasons for the project failing before the kickoff”

I think that those are called Risk Lists.

I just ran into this on the Public Sector DPE Team’s blog: Learn how to quickly convert your .NET Applications to Service Oriented Solutions w/NET 3.0? (Web Cast)

To me, it sounds like a serious downplaying of the ‘A’ part of SOA. Actually, I find this to be true around most of Microsoft’s “Service Orientation” message.

Remember how easy it was supposed to be to “convert” your apps from VB6 to .Net when it came out? Expect that to be peanuts compared to a move to SOA. Changing your app’s communication to WCF won’t improve your system’s architecture by itself, let alone transform it to become business aligned (ughh, how I hate these marchitecture type words).

If you haven’t done the deep business analysis before developing your app, don’t expect Indigo to magically make it happen, quickly or otherwise.

I guess that I’m just sick of Microsoft spinning everything as “easy” and “quick”. Nothing about our jobs are easy or quick, that’s why we get “the big bucks” and these projects cost an arm and a leg.

Arnon gave a nice writeup of one of the sessions at our recent meeting. The whole SOA thing came up again, but this time in the context of databases; so called Service Oriented Data Architectures (SODA). One of the points made in favour of SODA rested upon a thesis Jim Gray wrote a while ago entitled “Queues are databases”. Anyway, I won’t belabor the point seeing as Arnon handled that one.

What I do want to call out is Sql Server Service Broker’s hard-won feature of only-once and in-order delivery. From Arnon’s post:

“You can add readers to empty queues faster, and lock all related messages that are read by one of the readers (so you won’t have a situation where one reader reads the order and another reads the same order’s order-line before the first even updated the order into the database).”

And here’s the kicker. Modeling your messages in such a way is a rather poor design. Why on earth would you take an Aggregate Root like an Order, and split it up into multiple messages – orderlines and the order itself separate?

It cracks me up time and time again. Some new fangled product bends over backwards trying to support some really hard feature, which is then touted as a key strength, which, in turn, makes more developers work that way. If you design your messages properly, you won’t have any problem adding “readers” even without using Service Broker.

From the SOA in Action Blog, following up on a recent ZapThink Flash:

There has been considerable hand-wringing over the potential performance impact of SOA, and especially XML, on server environments. In a new think piece posted at Search WebServices, ZapThink’s Jason Bloomberg sheds some light on the challenge of high-performance SOA — and yes, he uses “high performance” and “SOA” in the same sentence.

First of all, SOA does not necessarily imply XML. Second of all, even if you use XML in server environment (I do, every day), while performance may be worse than binary serialization, I have yet to see it be the system bottleneck. Sending out unnecessarily large sets of data, or sending the same data over and over again (as occurs in an RPC world) has been problematic in many of the systems that I’ve seen. While one could blame XML for making the bloat worse, one might also blame the architect for the bloat in the first place.

“Granted, SOA is a services abstraction, and “every abstraction comes at a price,” Bloomberg observes. “Loose coupling, composability, agility, and the other benefits of SOA all introduce performance overhead.” At large sites with a lot of users and traffic, SOA performance is a huge challenge.”

From my experience, the price of the abstraction is negligible. Disregard for the impact of network latency comes with one helluva penalty, much higher than the service abstraction. Poorly designed services often suffer in this regard, so one could mistaken the performance hit as being caused by the abstraction. That is, if one was an analyst and not somebody actually doing the work.

One thing I will grant, though: “At large sites with a lot of users and traffic, … performance is a huge challenge.” Regardless of SOA.

Following my article in the Architecture Journal, I’ve been getting quite a stream of emails asking for more information. The latest one came from Alex and I’d like to relate his thoughts here, with my comments inserted inline.

“Udi,

1. You are saying “When services interaction is asynchronous we avoid
changing data in other services altogether”. The example you were discussing
is using the service B without changing its state so in this case no
transaction will have to include the service B. But if the data on service B
has to be modified I am not sure how this can be implemented asynchronously
(the way you define it) with publishing the state of service B to service A
and storing a subset of this state there. If the operation results in
changing the state of the service B, then how will using the local copy of
the data on service A accomplish the same?”

I’d like to address your first question first. “But if the data on service B
has to be modified I am not sure how this can be implemented asynchronously…” The answer is in the form of a question. Does the data in service B HAVE to be modified? What if we divided up the responsibilities between our services in such a way that all transactional behavior would be local to a single service; long running, inter-service workflows are not transactional end to end. This is one of the practical bits of guidance that gives you feedback on the quality of your service partititioning. This is essentially the answer to your second question too. The operation will not require changing the data in service B, by design.

“2. I think the pattern you were discussing should emphasize the reliability
of delivering the update notifications to the subscribers. In case of
synchronous call the error can be easily detected by the caller and handled
accordingly. In broadcasting model you were discussing the delivery of the
messages must be guarantied by the framework.”

Reliability is a big topic. I’ve posted on the topic before and you can find the summary here. There really are two parts to the reliability of notifications. Consider a smart client that is subscribed to a service which sends it stock updates once a second. When the user turns off his computer, should all those stock updates sent from then until he turns his computer on again be lost or not? What if it wasn’t a smart client, but a different service? What if wasn’t stock updates, but the results of certain actions requested earlier? It really does depend on the scenario.

However, the point that you bring up about handling errors when communication is fully asynchronous is a good one, and a topic for a separate post.

“3. It may be obvious but I think it is worth mentioning that a service
publishes only the items that changed their state (added, modified,
deleted).”

Again, I’m not sure such a broad statement would hold. Certain data is published in aggregates where, even if certain branches are unaffected, the entire aggregate is sent. This especially holds true for reference data that legacy systems may not have stored locally. Desigining a quality service contract is difficult, and goes hand in hand with properly assigning responsibility to various services.

Well, Alex, I hope that covers your questions. If something still is a bit unclear, feel free to fire me an email to Blog@UdiDahan.com.

After reading this article you just might be thinking that business rules management systems (BRMS’s) are the key to what SOA was supposed to bring (now that we’re in the trough of disillusionment) – that is, if you haven’t tried working with these beasts before.

There are several now-well-known fallacies around BRMS’s. One is that business analysts write the business rules, in plain English, and these rules do not require a programmer to enter/code them into the system; the BRMS handles all that.

“Costly, time-consuming transformation from business terms to programmer requirements and subsequent implementation can be reduced or eliminated. This in turn empowers business analysts to create and maintain the business logic directly, allowing them to make changes with little or no IT intervention.”

But, the author forgets to mention who performs these trivial activities:

“After testing and validation the business logic can be deployed to a business rules server.”

This brings me to the single most important thing anyone should know about the business rules approach. Employing a rules engine opens up the possibility that the system will behave non-deterministically. Consider that only 3 non-linear equations are needed to create chaotic behavior, and try to imagine what will happen in a system with hundreds and thousands of business rules that affect each other.

Yes, business rules often end up changing data which, in turn, causes other business rules to be activated. No, it is not humanly possible to look at such a set of rules and predict what will happen when a certain event occurs. You can fire such an event in a test environment and see what will happen. When the results aren’t what you wanted (that would be all the time), you can use the BRMS’s to show you which rules were run as a result of that event. Let’s see a business analyst debug that. “Plain English” my butt.

Let’s not go too much further on the BRMS on its own, but rather view it in the light of service-oriented architectures.

In the article, an approach is shown that is supposed to bring you the best of both SOA and the Business Rules Philosophy. I’m afraid that my experience has been otherwise. The author states:

“The business rules server may be implemented as a Web Service that is accessible to many SOA enabled applications across the organization. By supporting shared business logic within the SOA architecture that can be addressed by many disparate applications, organizations can reduce redundancy, speed implementation, lessen inconsistency and improve the overall efficiency of both the applications and the business processes they serve.”

This desire to concentrate all business logic into a single BRMS embodies the 11th fallacy of enterprise computing, as put forth by James Gosling “Business logic can and should be centralized”. Not to mention that this goes against the grain of SOA where each service is autonomous and is entirely responsible for all of its data and behavior. There’s another reason why the business rules community is pro centralization – the BRMS’s are so damned expensive.

Anyway, I don’t want to be all doom and gloom today so I’ll end on a positive note. While good OO practices and the use of the Domain Model Pattern will take you VERY far in making business rules explicit, and, therefore, their maintenance and evolution much less costly, sometimes you really neeed a BRMS. Just so you know, the investment you made in the domain model is not written off when you “upsize” to business rules – the rules themselves are written using the concepts exposed in the domain. As for the service, well, its message handlers keep on working the same as they always did; parsing messages and making calls on the domain model. The business rules will just respond to the events raised by the domain; message handlers don’t need to call into the BRMS.

Oh, and the next time somebody tells you how business analysts will be doing the work instead of programmers, ask them if they’re willing to take the debugging too 🙂

In this podcast, we describe the service oriented analysis process that leads to coherent services that are strongly business aligned. Verification techniques are also presented which can point to poor service decomposition. Finally, we drill down and show how services can be decomposed further into “Business Components” and “Autonomous Components”.

Resources:

1. Autonomous Services and Enterprise Entity Aggregation
2. Autonomous Services Podcast
3. Ask Udi a question on SOA
4. Subscribe to the Ask Udi podcast

Get it via the Dr. Dobb’s site here.

Or download directly here.

Want more? Go to the “Ask Udi” archives.