So you’re code threw an exception for some reason. Now, your WF workflow gets terminated. You’re not a happy camper. You go looking for help and find this practical set of solutions, but think to yourself “it shouldn’t be this hard”.

And you’re absolutely right.

The fact of the matter is, you don’t have to do anything special to deal with failures in workflow. It all can be handled in terms of messaging, since the only interesting kind of workflow is the “long-running” kind. Long running workflows are those that may involve calling out to other services/systems, or involve humans in the process. Conversely, “short-running” workflows involve receiving a single message, handling it by calling code on the same server, in the same process, and on the same thread, and possibly returning a response. Simple request/response interaction is short-running. Things you might think to use BizTalk for would probably be long-running.

Take a look at this post which shows that long-running workflow is just regular messaging with a sprinkle of state-management.

Anyway, in my article describing failure handling for messaging, these workflow failures fall under the following category and solution:

Other kinds of failure conditions include having our DB transactions fail because they were chosen as the victim of a deadlock. The appropriate system level error handling is to just retry the transaction a bit later. Once again, if we were using our queues within the context of a transaction scope that included that of the database, the message would return to the queue when that deadlock exception bubbled through. This is exactly the behavior that we want in this case, although having the message go to the end of the queue instead of the front would also be just fine.

Other kinds of failure conditions we might run into include things like unique-constraint violations, attempting to perform actions on entities that no longer exist, and other activities where trying them over again probably won’t yield any different behavior. In these cases, if an exception were to cause the perpetrating message to return to the input queue we’d get exactly the wrong behavior. At the very least we would want to maybe log the exception information

So, in some cases, it makes perfect sense for the workflow to let the exception bubble through. Not so that the workflow will be terminated, but rather that the message which triggered it would be tried again. You can configure Workflow Foundation to have this behavior as well, or so I’ve been told.

Now you’re happy, the problem’s been solved for you – by design. No need to go mucking around in the nooks and crannys of FaultHandlers, CallExternalMethodActivity, and hacking the InsertWorkflow stored procedure.

Clean, Correct, and Simple – courtesy of The Software Simplist.

Additional References:

• Ron Jacobs interviews me on SOA and Workflow
• A conversation on Workflow Foundation and Service Orientation

It appears that something of a consensus has been reached on the issue of Entity Services. So far I’ve been against them, describing them as a Common SOA Pitfall, among other things. Roger Wolter from Microsoft (and one of the brains behind SQL Service Broker) has also called out against them from a data perspective. But the recent back-and-forth I had with the pragmatic Nick Malik, from Microsoft’s Enterprise Architecture Group clinched it. Jack van Hoof also seems to be in agreement.

Entity Services are in the Application Architecture domain.
They are not a member of the Services which are the primary unit of decomposition of SOA.

An SOA Service may be implemented using Entity Services, or not. If so, those Entity Services are not allowed to be accessed outside that SOA Service’s boundary. As such, they are an implementation detail of SOA. In which case we can have very interesting discussions about the relative value of Entity Services over things like the Domain Model Pattern.

So, as Jack said: “No, entity services are not an anti-pattern.”

However, when you’re going to decide on the main Services that are going to be a part of your SOA, those Entity Services don’t belong there.

For more information, take a look at my article from the Microsoft Architecture Journal, only partially available by virtue of the Wayback Machine: Autonomous Services and Enterprise Entity Aggregation

Some Technical Difficulties

Ayende and I had an email conversation that started with me asking what would happen if I added an Order to a Customer’s “Orders” collection, when that collection was lazy loaded. My question was whether the addition of an element would result in NHibernate hitting the database to fill that collection. His answer was a simple “yes”. In the case where a customer can have many (millions) of Orders, that’s just not a feasible solution. The technical solution was simple – just define the Orders collection on the Customer as “inverse=true”, and then to save a new Order, just write:

Although it works, it’s not “DDD compliant” 🙂

In Ayende’s post Architecting for Performance he quoted a part of our email conversation. The conclusion I reached was that in order to design performant domain models, you need to know the kinds of data volumes you’re dealing with. It affects both internals and the API of the model – when can you assume cascade, and when not. It’s important to make these kinds of things explicit in the Domain Model’s API.

How do you make “transparent persistence” explicit?

The problem occurs around “transparent persistence”. If we were to assume that the Customer object added the Order object to its Orders collection, then we wouldn’t have to explicitly save orders it creates, so we would write service layer code like this:

On the other hand, if we designed our Domain Model around the million orders constraint, we would need to explicitly save the order, so we would write service layer code like this:

But the question remains, how do we communicate these guidelines to service layer developers from the Domain Model? There are a number of ways, but it’s important to decide on one and use it consistently. Performance and correctness require it.

Solution 1: Explicitness via Return Type

The first way is a little subtle, but you can do it with the return type of the “CreateOrder” method call. In the case where the Domain Model wishes to communicate that it handles transparent persistence by itself, have the method return “void”. Where the Domain Model wishes to communicate that it will not handle transparent persistence, have the method return the Order object created.

Another way to communicate the fact that an Order has been created that needs to be saved is with events. There are two sub-ways to do so:

Solution 2: Explicitness via Events on Domain Objects

The first is to just define the event on the customer object and have the service layer subscribe to it. It’s pretty clear that when the service layer receives a “OrderCreatedThatRequiresSaving” event, it should save the order passed in the event arguments.

The second realizes that the call to the customer object may come from some other domain object and that the service layer doesn’t necessarily know what can happen as the result of calling some method on the aggregate root. The change of state as the result of that method call may permeate the entire object graph. If each object in the graph raises its own events, its calling object will have to propagate that event to its parent – resulting in defining the same events in multiple places, and each object being aware of all things possible with its great-grandchild objects. That is clearly bad.

What [ThreadStatic] is for

So, the solution is to use thread-static events.

[Sidebar] Thread-static events are just static events defined on a static class, where each event has the ThreadStaticAttribute applied to it. This attribute is important for server-side scenarios where multiple threads will be running through the Domain Model at the same time. The easiest thread-safe way to use static data is to apply the ThreadStaticAttribute.

Solution 3: Explicitness via Static Events

Each object raises the appropriate static event according to its logic. In our example, Customer would call:

And the service layer would write:

The advantage of this solution is that it requires minimal knowledge of the Domain Model for the Service Layer to correctly work with it. It also communicates that anything that doesn’t raise an event will be persisted transparently behind the appropriate root object.

Statics and Testability

I know that many of you are wondering if I am really advocating the use of statics. The problem with most static classes is that they hurt testability because they are difficult to mock out. Often statics are used as Facades to hide some technological implementation detail. In this case, the static class is an inherent part of the Domain Model and does not serve as a Facade for anything.

When it comes to testing the Domain Model, we don’t have to mock anything out since the Domain Model is independent of all other concerns. This leaves us with unit testing at the single Domain Class level, which is pretty useless unless we’re TDD-ing the design of the Domain Model, in which case we’ll still be fiddling around with a bunch of classes at a time. Domain Models are best tested using State-Based Testing; get the objects into a given state, call a method on one of them, assert the resulting state. The static events don’t impede that kind of testing at all.

What if we used Injection instead of Statics?

Also, you’ll find that each Service Layer class will need to subscribe to all the Domain Model’s events, something that is easily handled by a base class. I will state that I have tried doing this without a static class, and injecting that singleton object into the Service Layer classes, and in that setter having them subscribe to its events. This was also pulled into a base class. The main difference was that the Dependency Injection solution required injecting that object into Domain Objects as well. Personally, I’m against injection for domain objects. So all in all, the static solution comes with less overhead than that based on injection.

Summary

In summary, beyond the “technical basics” of being aware of your data volumes and designing your Domain Model to handle each use case performantly, I’ve found these techniques useful for designing its API as well as communicating my intent around persistence transparency. So give it a try. I’d be grateful to hear your thoughts on the matter as well as what else you’ve found that works.

Related posts:

• Fetching Strategy Design – showing how to separate the concern of eager loading from both your Domain Model and your Service Layer.
• Better Domain-Driven Design Implementation – showing the basics of how valuable interfaces between your Domain Model and the Service Layer can be.
• Lazy Loading, and how messaging fixes everything again – describing the advantage of O/R mapping your message classes as well.
• Query Objects vs Methods on a Repository – discussing the scalability (in terms of number of developers and queries) of Query Objects.

Jimmy’s recent post called out some of the insights on the advantages of Ruby from Niclas’ keynote at DevSummit 2007. Jimmy writes:

In Ruby it’s easy to redefine the visibility of a method from private to public for testing purposes. This was just one small detail in his talk of course, but I started to think about how much pain I quite often go through regarding this in legacy code.

Let me just start out by saying that dealing with legacy code is far from easy. However, when writing new code, or as a part of refactoring that legacy, adding interfaces to your design can help you get around those visibility and testing issues.

For instance, if you only let “client” code access the interface, and not the implementation (probably using some kind of dependency injection), then you could leave all sorts of methods public on your concrete class without worry that the client will call them since the interface doesn’t expose them. Now that the methods on the concrete class are public, you can easily test that class.

The way to package your code to make sure this occurs follows a very simple design principle. This is much easier to put in practice on new development, but you’ll find that it isn’t that hard for legacy either. Since legacy code often doesn’t make use of interfaces, and the implementation is already packaged, and your new client code will already be separately packaged, you’re 90% there!

If you can change the legacy code, you’re actually 95% there. Just create a new package for the interface that the client code will use. Then change the legacy code to implement that interface – since the methods will be pretty much the same, it’s not that much work.

If you can’t change the legacy code, create a “wrapper” class that implements the interface and delegates the calls to the legacy code.

Finally, let me sum up by saying that I think Ruby is great. However, I think that often many advantages are attributed to anything new that may have already been possible/easy with what we already have today. Sometimes, the new stuff helps raise awareness on important issues – and then we can have great discussions on how to solve those issues with today’s (yesterday’s ?) technology 🙂

My good friends Eyal Vardi and Ohad Israeli, the guys behind Experts 4D are putting together one heck of a conference this July in Israel: R U Ready 2008.

Although the timetable isn’t set in stone yet, I’m going to be doing a full day course on July 2nd, so mark your calendars. The official link is here, but I’ve got the full information below. Here’s the registration page.

By the way, if you’re interested in having me give this course to your team, give me a shout.

Course Description:

Designing large-scale distributed systems is hard. New technologies make it easier to comply with today’s communications and security standards, but don’t auto-magically give you a robust and scalable system. Join Udi for a day packed with the wisdom of companies like SUN, Amazon, and EBay. Tried-and-true theories and fallacies will be shown, keeping you from making those same costly mistakes today. Communications patterns like publish/subscribe and correlated one-way request/response will be used in conjunction with advanced object-oriented state management practices for long-running workflows. If you enjoy deep architectural discussion, if you are in charge of building a large-scale distributed system, if you want to know more about how the big guys run their systems, this day is for you.

Course Content:

Module 1: Distributed Systems Theory

Decades of distributed systems development have taught us many lessons. In this module we’ll cover many historical mistakes as well as proven best practices for scalable and robust design. Topics include:

• 8 fallacies of distributed systems
• Transactions

Module 2: Identifying Coupling in Distributed Systems

Loose coupling has become the watchword of complex systems development, yet few understand its multiple dimensions. In the module we’ll be covering the three different dimensions of coupling as well as patterns for dealing with them.

• Platform Coupling – XML/SOAP
• Temporal Coupling – Synchronous/Asynchronous
• Spatial Coupling – Endpoints/Topics

Module 3: Asynchronous Messaging Patterns

Although scalability is achieved through the use of asynchronous message passing, more advanced message exchange patterns are required to handle today’s complex integration scenarios. This module will cover the most commonly used patterns:

• One way
• Correlated Request/Response
• Publish/Subscribe

Module 4: Bus and Broker Architectural Styles

Enterprise Service Buses are all the rage these days. In this module we’ll be covering what’s the difference between the Bus architectural style, and the more well-known Broker, found commonly in many EAI projects. Topics will include:

• Architectural advantages and disadvantages
• Technological advantages and disadvantages

Module 5: Long running work flows

The distributed communications patterns wouldn’t be complete without a discussion on orchestration. In this module we’ll see how to manage the state of long-running distributed communication flows as well as:

• How to encapsulate work flow logic
• Advantages and disadvantages of orchestration

If you’re interested in the full 2-day course which includes SOA topics, check it out. Advanced Distributed Systems Design with SOA – learn from the mistakes of others, succeed in your SOA project.

After seeing Mark’s post on Reasons for Isolation describing the ways Layered Architectures break down, and the ways making it more testable can change it, I’ve got to wonder – is Layering just too simplistic to actually work?

Just the other day I was doing a design review for a fairly simple Smart Client whose design was layered. In order to stay away from interfaces that accepted dozens of ints, strings, and dates, they wanted to have each layer talk to the other using “entities”. So where are these entities defined – oh, in a “vertical layer” that all the horizontal layers talk to.

OK, so we’ve taken the simplistic one-dimensional layered architecture and added a dimension. What now?

Well, it seems that having the business logic and the entities in separate layers goes against one of the most basic Object Oriented principles – encapsulation. So, let’s put the entities back in the Business Logic Layer. But then how will the Data Access Layer accept those objects as parameters?

So, that is solved by keeping Entity Interfaces in the “vertical” shared “layer”, and having the entities in the business logic layer implement those interfaces. That way, the data access layer can still accept parameters corresponding to those interfaces:

void InsertCustomer(Shared.Entities.ICustomer customer);

So far so good. Now, we want more testable UI layer code – so we use Model-View-Controller (MVC) – of whichever flavor suits your fancy. I’d say that Supervising Controller is a must. You could also add another presenter for more complex screens as in Passive View, but I’d be less strict on that. So, in which layer do these Controllers/Presenters sit? And is the Business Logic Layer the Model? Or is the Model just part of it?

Well, our Supervising Controllers are those who decide what action to do and when, where to get the data from, etc. That sounds like business logic to me. So let’s put them in the BLL. Presenters for the Passive View are much more UI centered, so let’s put them in the Presentation Layer. But we don’t want them tied to the implementation of the view, so we’ll put them in a separate package, and have them depend only on the view’s interface. So we’ll put the view interfaces in a package separate from the view implementation as well.

If it wasn’t clear up to this point, all the questions raised in this post are architectural in nature – as in they have a substantial impact on the structure and flow of the system, and will definitely have a profound effect on its maintainability. In other words, if you think that Layer Diagram covers your design – you’re probably deluding yourself. Personally, I think that’s why many developers consider architects to be “out of touch with the real world”.

When you have a design that answers these, and other architectural concerns, you’ll find that layering is of little importance. The specific constraints on each package are what counts. The fact that the Presentation Layer can talk to the Business Logic Layer doesn’t mean that the classes in your Views Implementation Package can. A large part of an architects work is to specify these constraints, and communicate them to the team. Tools like FxCop may help in terms of enforcing these constraints, but I believe that getting the team to actually “buy-in” is more effective.

Single-dimensional layered architectures don’t work. They violate Einstein’s maxim:

Make everything as simple as possible, but not simpler.

Layering – “simpler” to the point of simplistic.

In this podcast we answer questions about how to use autonomous components to unify disparate building blocks like servers, middleware, and databases in order to handle the technical complexity of complying with detailed service-level agreements. Reuse of business logic, database schemas, and messaging topics between autonomous components are discussed as well.

Download via the Dr. Dobbs’ site.

Or download directly here.

And here’s this week’s question:

Hi Udi,

Thanks again for your continued assistance. I was very much interested by your advice to consolidate each of the services related to each product family into a single service, but as autonomous components.

From your description of autonomous components from a prior podcast, it seems that they are much the same as services – in that they communicate only via loosely coupled messaging, and can have their own databases. Would you say that the main difference between autonomous components is that different autonomous components within a service may in fact share business logic and databases? If so, it would seem that combining these services into a single service with 3 autonomous components would be a matter of definition, rather than an architectural shift. Any information you could provide to clarify this distinction would be fantastic.

Something else that’s been playing on my mind of late – is whether or not you would consider a topic as having to belong to a specific service. That is, would you say it is bad practice to have multiple services publish on a common topic? I suppose if we have multiple services publishing on a common topic, then they should be defined as autonomous components, belonging to a single larger service – in which case that common topic would belong to that new service.

As usual your advice is always extremely helpful. Please keep those podcasts coming!

Best Regards,
Bill

Additional References

• Podcast on Message Schemas between multiple Publishers and Subscribers
• Podcast on Business and Autonomous Components in SOA

In this podcast we answer questions about how to solve dependencies between systems that subscribe to events in SOA. We’ll also get into the pitfalls of employing distributed transaction when reusing existing systems even behind service boundaries.

Download via the Dr. Dobbs’ site.

Or download directly here.

And the original question was:

Udi:

I have a question regarding publishing events that relate to data changes. I found the article you wrote in the Arch Journal #8 very helpful. I think striving for autonomy is very important. The scenario I was thinking about is how can you ensure synchronization across subscribers of a particular event.

For example, System A publishes an event when customer information is updated. There are several systems that subscribe to this event. Two of the systems, System B and System C, need to be sync regarding customer information. System B uses operations from System C using the customer data. Using your example, System B has a process that runs for all “Preferred Customers”, and it uses processes on System C. However, System C may not have process the event to and may have a customer as preferred.

I have several thoughts, but would like to get your thoughts on this scenario. Are there any best practices or patterns?

Phil

Additional References:

• Autonomous Services and Enterprise Entity Aggregation (Article in Microsoft Architecture Journal)
• Podcast on Autonomous Services and Pub/Sub

You can find more episodes like this in the Ask Udi archives.

The article I wrote a while back, The Enterprise Service Bus and your SOA has been translated in to German and has been published as a part of Stefan Tilkov (of InnoQ fame) and Gernot Starke book: SOA-Expertenwissen.

I now have a German bio!

(Take a look at some of the other articles I’ve written, as well as other books I’ve contributed to)

I’ll also be doing a webcast for UML China, “the most well known website on OO/UML for software developers in China”. I’ll be proud to be joining the distinguished list of presenters for the site, including names such as Eric Evans, Jimmy Nilsson, Martin Fowler, Alistair Cockburn, Kent Beck, Scott W. Ambler, John Vlissides, and more.

I’ll be talking about the content I published in the Applying Domain-Driven Design and Patterns book on SOA Basics.

Since I don’t speak Chinese, we’ll be doing it all in English so feel free to join in. It’ll be on June 13th, from 9 to 11 am, in the GMT+3:00 time zone. I’ll have the link up shortly. There will be a one hour presentation followed by a full hour of Q & A. Hope to see you there.

You can now download the PDF of the presentation I gave at DevTeach on What to Test, and When. My apologies for putting this up so late.

In brief, you can’t provide business value if you can’t ship. Shipping software means more than having some features passing some automated tests, although most of the discussions I’ve been hearing in the Agile circles seems to say that that is enough. Are you really going to be putting credit card numbers into that system before you’ve done any security testing? I hope not. Same thing goes for usability, scalability, etc.

So do you really have to be able to release software every two weeks, or month?

Bottom line, understand the terms and conditions of every release, and plan accordingly – both in development effort as well as in testing scope and amount. Also, sponsor-driven constraints like trade-shows will affect what you need to test and when.

So, viewing testing outside the context of hands-on project management is ineffective.

Why do we have that independent QA department again?