When developing systems that handle messages that arrive on queues or other asynchronous transport mechanisms, broader failure scenarios need can be handled than in RPC-style communications. One of these failure scenarios that’s been getting some recent play on the blogosphere is that of poison messages. Microsoft’s Nicholas Allen does a great job describing what defines a poison message and the common ways of handling it. Thomas Restrepo adds another facet to the whole issue of poison message handling that is a must-read.

Let’s take a rather simple case, but definitely a common one when we start versioning our services – that of the receipt of a malformed message. It could be that we were unable to deserialize the data or something more complicated, but at the most basic level, there’s nothing the service can do with the message. This is often expressed in terms of some sort of exception that reaches all the way down to the messaging infrastructure. If we were using the queue in a transaction, we’d see a rollback and the message return to the queue, causing the service to loop endlessly on that message. This sort of failure case is best handled by having the messaging infrastructure just move the message to an error queue.

Other kinds of failure conditions include having our DB transactions fail because they were chosen as the victim of a deadlock. Matts does a good job of describing why these things occur and how to handle and minimize their occurrence. The appropriate system level error handling is to just retry the transaction a bit later. Once again, if we were using our queues within the context of a transaction scope that included that of the database, the message would return to the queue when that deadlock exception bubbled through. This is exactly the behavior that we want in this case, although having the message go to the end of the queue instead of the front would also be just fine.

Other kinds of failure conditions we might run into include things like unique-constraint violations, attempting to perform actions on entities that no longer exist, and other activities where trying them over again probably won’t yield any different behavior. In these cases, if an exception were to cause the perpetrating message to return to the input queue we’d get exactly the wrong behavior. At the very least we would want to maybe log the exception information, but putting the message in the error queue doesn’t seem right. I mean, there was nothing wrong with the message, it’s just that the application state had moved on or that some other logic failed. One of the reasons for putting the message in the error queue is so that a person can manually fix what’s wrong with the message and send it back to the input queue for reprocessing. In these failure conditions, that hardly seems likely.

What I usually suggest for handling failures when messaging is involved goes something like this. First of all, have an error queue/topic – they’re exceedingly useful. Second, don’t automatically roll messages back. If the service wants to retry, they should do so after handling all the other messages already in the queue – unless (!) you’ve got message ordering issues like Thomas mentioned. This means having the service request the messaging infrastructure send the message back to itself. When it comes to exceptions around the logic handling the message, just write them to a log and be done with it. The service logic will probably log anything else that’s of particular interest to it. This leaves us with the malformed message scenario. In that case, the communications layer can know that the problem occurred before any service logic (like deserialization exceptions) and therefore can (rightly) pass the message along to the error queue.

Of course, for the above guidance to be valid, we need to educate service logic developers as to when it is appropriate for them to throw exceptions and when they actually have to send the message back themselves. It is often the case that by taking on certain constraints, we can greatly simplify the handling of scenarios which are difficult in the general case.

Ram asks if SOA can be implemented without web services, which seems to be a valid question once we’ve understood that web services do not equal SOA. He hones this question on the issues of homogeneity and interoperability. Given that architecture, the ‘A’ of SOA is technology independent, and in his post Ram even gives examples where an SOA is implemented on a single technology, maybe the question should be should SOA be implemented without web services. Or, in other words, from a project/risk standpoint does it make sense not to use web services?

The bottom line of Ram’s post is that “web services are essential to implement SOA if the environment changes to a heterogeneous one…”. I guess that the question is what actually constitutes a web service? Is SOAP enough? If so, are we talking about SOAP 1.0, 1.1, or 1.2? If not, do we also need more of the WS specs? If so, which ones? WS-Reliable Messaging? WS-Atomic Transaction? WS-Addressing? WS-Topics? That’s what’s so great about standards – there are so many to choose from. Of course the different vendors support different subsets of all these standards so interoperability is still something of a crap shoot.

I actually want to take this discussion in a slightly different direction. At the message-payload level, interoperability is handled well enough by XML and XSD, although RelaxNG looks so much more elegant for schemas. The question now becomes one of communication – what does a message look like and how are various communications patterns represented. At the simplest level, we have HTTP which is interoperable across (almost?) all platforms but also quite lacking in higher level features. Going up we see things like JMS, queuing, and other middleware. We actually have a reasonably good ability to stitch these together across platforms. In my opinion, this would be a good place to start in terms of enterprise, and other reasonably complex systems. Going even higher we get into things like spaces which support other architectural styles than SOA, have a richer feature-set, and are often much more expensive.

One thing I think that bridges many of the concerns here is to create an abstraction layer between your service logic and the communications infrastructure. This abstraction layer would be implemented in the same technology as the service. When using platforms that support things like interfaces (.net, Java, C++ pure virtual classes) this often yields two actual layers – one for the interface, and another for the mapping to the specific technology. Add to that the use of dependency injection and you can evolve your communications with the specs and platforms. You should expect to have such an abstraction layer for each of the technologies found among your services.

Once again, from a risk/project perspective this often yields the best of both worlds. We can start with a single platform keeping things simple, say a JMS, but remaining decoupled from it we are able to evolve with the technology while maintaining most of the investments made in our services. I think that that is one aspect of the agility expected from SOA.

On the projects I consult on, web services are really just an implementation detail and not a primary architectural concern. The communications interface is well defined logically and the mapping to the chosen technology is sometimes simple, and sometimes complex – say if the technology doesn’t support pub/sub and we have to implement it ourselves in the mapping layer. If we use WS-Addressing or WS-Topics really isn’t relevant. Other specs like WS-Atomic Transaction is something that we’ll likely never use since flowing transactions between services ruins their autonomy.

Anyway, my bottom line here is that SOA can be implemented without web services in both homogeneous and heterogeneous environments, and each project needs to analyze for itself whether or not it should. However, this technology question should in no way impact any of the architectural decisions made – it’s a communications issue. Also, be aware that not all inter-service interactions should be the same, some need reliability, others need to move tons of messages, others need to interoperate well with external partners. Plan and schedule for these special cases, they take time to get right and no WS spec will magically make everything alright.

Regu posted an interesting question recently: “Is scalability a factor of the number of machines/CPUs?”. His answer can ultimately be summed up as “yes, but…” – it was qualified in terms of threads: “… scalability in a well designed system is a factor of number of threads that can be efficiently executed in parallel”. The word “efficiently” meaning that the threads are actually doing work and not just waiting. However, the question of how many machines do we need is a hard one. Nick calls out a very important point on this, “An asymmetric farm, with machines of varying capabilities, is really hard to tune.” In all cases we find that load-leveling mechanisms like queues are good for scalability.

Just as a slight sidebar for anybody who deals with systems where work needs to be divided up and run in parallel to achieve required latency requirements, we have to deal with all the above problems and more. For instance, if we have to process images, finishing the processing on each image in one minute. Now, we have an algorithm that can do part of an image that runs at a speed of 1MB per second, single threaded on a dedicated machine with a standard 3GHz processor. So, how can we process a 1GB image in 60 seconds? Simple, get 17 processors right? Well, if you were running a 16 or 32 way SMP machine then probably yes. But what if you want to scale out, say, because you’re receiving one image every 2 seconds on average? Well, once we scale out, time is impacted quite significantly by the cost of just moving data between servers – one of the fallacies of distributed computing. It becomes a much more difficult problem – the kind that I just love sinking my teeth into 🙂

Anyway, a lot of us aren’t dealing in these massively parallel problem spaces but are just looking for good scalability advice. Well, one of the characteristics of a scalable system is that load is evenly distributed between machines (up to a point – if we have more machines than work that needs to be done, some will be idle). Load can be broken up in terms of resource usage – CPU, memory, disk, network, etc and we should be looking at all parameters. I’ve noticed a tendency of people to focus only on CPU usage. One case I consulted on was a system that was having performance problems although average CPU utilization was around 50%. They did a costly hardware upgrade at the time from single-CPU machines to all double-CPU, hoping to drive down the utilization and improve performance. They only succeeded half way – CPU utilization did drop, but performance (in terms of response time and throughput) didn’t improve – quite simply because the network was the bottleneck, and not processor power. As Dan so eloquently states: “Latency exists, Cope!”

If you use the Pipeline architectural pattern (page 5) that is so well known in the embedded/real-time space at the macro level (inside the service, not between services – that’s SOA), and SEDA (Staged Event-Driven Architecture) at the micro level you can create an environment where you can know the amount of resources you need to buy/provision for the expected load at a high degree of accuracy. An additional, maybe even more important benefit has to do with the resiliency of such a system. If there is a degradation in resource performance or availability, the system won’t come crashing down but rather “limp along”. Conversely, if load continues to increase beyond expected maxima, the performance (in terms of throughput) of such a system would not degrade. By monitoring response time per request, you could notice the upward trend and provision more resources. If you were working with a grid-like infrastructure, you could set these rules up so that they would be executed automatically. These are the building blocks for building “self healing” systems – one of my current favorite areas of interest.

Bottom line, I’ve found that the layered-architecture/tiered-distribution pair to be rather limited in terms of scalability (in terms of load). I would say that the solution isn’t necessarily to move to a Space-Based Architecture, as Guy mentions in this post, although many of the event-based concepts are definitely broadly applicable. Werners Vogels (Amazon’s CTO) mentions the CAP (consistency, availability, partitioning – choose 2) model for distributed systems in this podcast which I think is critical in analyzing the different parts of a complex system. On the flip side, Patrick does an excellent job of warning about the dangers of other appealing, siren-esque paths – follow them at your peril.

I’m afraid that there aren’t any easy answers, but at least we have some models that have proven themselves viable in the most strenuous scenarios. These models sometimes contradict popular architectural styles and it’s good to be aware of that. At the end of the day, it is our job to make the difficult technical tradeoffs.

After I finished reading Arnon’s SOA definitions post I felt a distinctive distaste for many of the common terms in the industry’s SOA vocabulary. Let’s say that communications between my services occurs over a pub/sub channel – a topic. One service publishes messages on that channel, another receives them via its subscription. Let’s go over the SOA terms in this context. 

Contract: Who owns the message type being published? The publisher or the subscriber? Common SOA knowledge would say that the message belongs to the contract of the service that receives it. However, would that receiver “control” that message type? Would it be in charge of versioning it? I would put my money on the publisher of that message type. I think that the concept of contract is more far-reaching than just which messages a service receives. Rather, contract seems to be tied quite closely to the business-level responsibilities of the service. This brings me to my next point: 

Endpoint: From Arnon’s post, “… a specific place where the service can be found and consumed. A specific contract can be exposed at a specific endpoint.” A service could probably have more than one endpoint, and it would make sense that not necessarily all of the contract in its entirety would be exposed at each one. But what about the topic described above? Is it an endpoint? If so, does it belong to the receiving service(s) or the publisher? It doesn’t make sense to have an endpoint that is shared between services, does it, or maybe that’s how we define service consumers? 

Service Consumer: “A service doesn’t mean much if there isn’t someone/something in the world that uses it.” Is the publishing service “using” the subscriber when it publishes a message? I don’t think so, and the subscriber definitely isn’t using the publisher at that point either. So, we’ve got some inter-service message-based communication going on and it isn’t clear if we even have a service consumer. In fact, if all a service ever did was subscribe to some topics, and publish messages on other topics, it looks like we’d have very loose-coupling but be straying from the common SOA wisdom.

And I guess that that’s my bottom line. Patterns for building loosely-coupled, large-scale systems existed prior to the SOA tagline, and SOA (or EDA, or whatever TLA you want) has come to stand for those very patterns. However, somewhere along the line vendors appear to have gotten hold of the discussion around SOA and have apparently polluted it with terms that only cloud the original message. I know that Arnon agrees with me on many of these points (seeing as we’ve done some projects together according to these exact principles) so I don’t want this to come off the wrong way. But we, as an industry, really have to get back to our roots here, because I see this new vocabulary steering us in all sorts of sub-optimal directions.

Nick talks about different options on how to modernize legacy applications for SOA environments and it reminded me about one of the common SOA anti-patterns I see. First of all, I am all for capitalizing on legacy, which has been called “code that works”. SOA is not necessarily a rip-and-replace philosophy. However, just wrapping up a legacy app in a bunch of web services will probably not bring you much closer to the promised value of SOA.

Let’s say you have a mainframe running CICS which controls much of your transaction processing logic, but none of the newer web-enabled stuff – that’s being done on Windows boxes running IIS and .net. The anti-pattern that I’ve been seeing here, there, and I hope it isn’t everywhere yet goes something like this. Put a web service layer in front of the mainframe and another in front of the .net logic on the Windows boxes, and orchestrate them all with some kind of engine. While providing interoperability, one can hardly say there was any architectural foresight involved.

The alternative isn’t to throw away all the investment made on either the mainframe or Windows side and rewrite it in some other technology – after all, we’ve got a business to run. After iterating through the appropriate business and architectural analysis to identify what our top level services are and what messages flow between them, we look at implementing them using our existing systems. What this means is that our mainframe may be involved in multiple services, and that it stops being a top-level architectural element. 

Over time we can look at better back-end separation within the mainframe – decreasing the coupling between the various parts involved in different services. Eventually, when we begin discussing the business value provided by a given service, we find out that the cost of running it on the mainframe justifies a rewrite, so we do that. By balancing operational cost, time to implement changes, and per-request business value we can make educated decisions around how to best employ and evolve our legacy while maintaining business continuity.

While such efforts will take time, the ability to keep business running and make incremental improvements to the areas that will provide the highest value, all the while moving in the right direction strategically, that’s a win-win scenario all around.

I’ve finally gotten around to looking at the last installment in my ongoing conversation with Bill. This time he corners me on Request/Response:

“Hi Udi, Thank you for your response. I actually came to the same conclusion as you eventually. I am now representing each POS “station” as part of the same “sales” service. This is coming along quite nicely. I still however have one service remaining that is requiring a request-reply message exchange – and that is the Risk Assessment service.

The Risk Assessment Service accepts an “assess risk” message containing a risk profile and then returns a risk assessment containing a series of decisions made by the service.

A service (as opposed to just a library) is appropriate here because the rules, rate charts and risk point charts are all housed in a central database. Moreover, the Risk Assessment view of the world is different to the Policy Administration view, Sales view, Finance view, etc – so there is a lot of benefit to be had in fracturing the domain along these lines and creating a separate service. Risk profiles that are deemed high risk are placed in a queue where they are manually reviewed by an Underwriter before being approved or declined.

Other services (Sales and Policy Administration) must leverage the Risk Assessment service to fulfil their own responsibilities. I can’t see any way of getting around a request-reply exchange in this circumstance.

What are your thoughts on this? Once again thank you for your valuable time and insight.

Regards, Bill”

Well, Bill, it looks like Risk Assessment is a good choice for a service in your system. There is low coupling at the business level and it doesn’t appear that there is any need to have transactions cross its boundary.

I have to admit that it sounds like you’ve got a case where Request/Response would fit fine. That said, I’d still keep it asynchronous – which seems to pretty much be a constraint you can’t get around once you have people in the loop (the underwriters).

The messaging pattern which describes this is Correlated Request/Response, also known as Duplex in Microsoft’s WCF stack. What this means practically is that, on top of the message Id in the header, there is also a Correlation Id. The value of the Correlation Id in the response message is the value of the Id in the request message.

You might want to consider using the Return Address pattern as well, where the response message isn’t necessarily sent to the source of the request message, but rather to a location specified in the header of the request message. This enables the requesting service to split its load into two: one for the requests it handles itself, another for the responses it receives – a big plus in terms of scalability.

But don’t write off pub/sub just yet! You could still have your Risk Assessment service publish a RiskAssessmentCompletedMessage and have the subscribers filter out the ones they want – either based on the Correlation Id or some other application-level identifier. The pattern which describes this is a variation on Correlated Request/Response and is called Command/Notify. If you have a very large payload that you wouldn’t want clogging up your network, you could possibly write that to a central storage area and put its URI in the message, REST style.

Although you could “splice” an endpoint so that you can audit the messages it receives, it is much easier to instrument pub/sub channels. Large, distributed systems are hard to get right so instrumentation is important. Anyway, I hope that helps – and “better late than never” 🙂

The issue of concurrency is one of particular interest to me. I’m not sure if it’s because I have seen too many projects get into serious trouble around it, or that I’m just weird that way. After these last few days at Arosa at the Software Architecture Workshop, I’m beginning to understand that there’s a pretty good chance that I’m just weird. I mean, it turns out that I do quite a few things different from most other people. So, after reading Matts’ great post on “Realistic Concurrency”, even though it was after midnight, I had to get my head around his thinking.
 

After framing the issue as a business-level update of information in a multi-user system, we drilled into the specifics. First of all, we agreed that pessimistic locking in business terms meant treating that part of the system as single user for when the user had acquired the lock. The pattern that is often used in that case is what I call the “Make it so” pattern, after Picard from Star Trek, The Next Generation. In this case, the user sets up the data the way he wants and tells the system to “make it so”. From a software perspective, that data snapshot can be sent to the server without very much fanfare since no other user could work on the data in the meantime. It has been my experience that projects usually get these parts right.
 

Before going into the optimistic concurrency topic, I just wanted to mention that the way the system enables the user to acquire a pessimistic lock itself involves optimistic locking, since there is no way to prevent multiple users from attempting to acquire the same lock at the same time. I consider that somewhat ironic as I’ve seen numerous projects work under the premise that they could do only pessimistic locking and ignore optimistic concurrency.
 

Well, the main thing that I wanted to hear from Matts after reading his article was how was it that I had never run into the problems that he had described. After going back and forth about it a little, Matts proclaimed that the way that I was working was not usual. Maybe I am a bit quirky, but I find nothing special about this rather generic way of working:
 

1. Server gets a message
   
2. Service layer opens a transaction
   
3. Service layer requests the relevant object (often by ID)
   
4. Service layer calls a method on that object
   
5. If no exception is thrown, commit the transaction, otherwise rollback
   

Matts put me to task by asking exactly what kind of method would I call for a generic update. I have to admit that that was not something I was prepared for. A generic update? In a business-level optimistic concurrency scenario? I don’t have any of those. Its all specific – change customer address, cancel order, authorize payment, etc. This specific information is passed as a part of the message, including the relevant data, which in turn get passed to the specific method. The transactional scope makes sure that if two concurrent actions are received for the same instance, they will be performed serially. If there’s any validation to do, it’s done as a part of the specific method.
 

And then I saw Matts eyes light up; “you’re doing exactly what I’m suggesting! Before going to update the data, you refresh it.”
 

Apparently I was missing something; “I didn’t know there was any other way to do it. Its always just get the object, call a method. For example, canceling an order: you have to check if the order hasn’t been shipped yet. How could you possibly do that if you didn’t get the current state of the object? I don’t think I understand the alternative.”
 

Weird, dense, or tired, but Matts got me to understand in the end. The underlying premise that I had was that the UI was task based for all things where multiple users could be working on the same data. After I understood that not all systems were like that, quite often editing data in a grid, I began to realize the root causes he was talking about. Matts had a much broader perspective than I in these matters because he wrote, and supports an object-relational mapper. He interacts with a much larger and more diverse type of projects and was trying to help solve problems in the general case. My suggestion was that this specific problem could be better solved by redesigning the UI. Users like tasks based interfaces – they correspond very well to the environment in which they operate. When a customer talks to a call-center worker, they don’t talk in terms of “I want to generically update my information”, but rather in terms of specific tasks: “I want to change my address, and cancel an order.”
 

This isn’t so much of a big deal in call center apps, since that user won’t often be doing similar tasks online concurrently with their conversation, but collaboration type systems are much more susceptible. Document management type systems are one example.
 

Well, by the time we were done yapping, it was already after 2 am but we had made some progress. We had recognized that no system could be entirely based on pessimistic locking, and the different patterns for each scenario. I was practically passed out at that point, but that was definitely a productive night, rounding out an amazing day.

Published in the IBM Portal of DevX.

Summary: Udi’s developer therapy group wrangles over the real advantages and tradeoffs in developing using SOA in the real world—especially compared to past practices.

Continue reading.

I’ve began rethinking certain assumptions about how I use message passing in my distributed systems after reading things like “SBA & EDA Lessons Learned” from the excellent “Panic from Fuzzy” blog. Since I was only familiar with the Tuple Space theory but have never employed space technologies like JavaSpaces, I did the only thing I could do – I convened a session on it at the Software Architecture Workshop so as to benefit from the knowledge and experience of those frightfully smart guys.
 The functional issue that attracted me to spaces was its ability to do very dynamic like content-based filtering. By subscribing to notifications based on a template, I could replace a broad and deep topic hierarchy and handle some other interesting scenarios as well. For instance, when a given user is working on a certain set of data – a tree of specific instances, I would like that machine to only get updates on that data, which could be quite a substantial savings in terms of network load. Another case is where the user is only allowed to see certain instances of the same type.
 

About three quarters of the way through the session, when I was still thinking that I could put entities in the space, someone called my attention to the fact that I would have to give up the cross-entity transactional semantics that I was so used to. For instance, in order to implement a business rule when an order is cancelled the customer may also need to be updated to a non-preferred status. This calls for a transactional scope around both of these updates – on the business level. If the entities were in the space, the naïve solution would be to Take the order, change its status to cancelled, Put it back in the space, Take the customer, updates its status, and Put it back in the space. By not being able to perform all this work transactionally, failure cases would need to be handled by compensating transactions – a significant jump in complexity.
 Needless to say I left that session quite a bit cooler on Space-Based Architectures than I was going in, but later on in the day, over a beer with one of the guys, he suggested that maybe spaces could be used as an alternative way to do message passing. Instead of Taking and Putting entities, we could do the same for messages – since the business-type of transactions would by-and-large correlate to a single message. We could benefit from the dynamic subscription behavior and possibly do away with a deep, and sometimes fragile, topic hierarchy. Once again my enamourment was on the rise.
 

But as do all fickle loves, this too was not destined to last. After doing some more thinking I remembered about handling failure cases in terms of message passing. When dealing with reliable messages, the receipt of a message, its handling, and the subsequent sending of new messages is at times required to be all transactional. I was once again required to perform a Take operation as well as multiple Put operations in a single transaction.
 I hardly believe that this is the end of the story for me and spaces. I’m still getting my head around using spaces as a technology choice in my current architectures and examining new architectural possibilities around it as well. We are definitely living in interesting times.

Update

After reading “What about Usability”, Jeremy’s latest installment of his “Better Software Development” series (my words, not his) I got flooded with a bunch of thoughts. Mostly things that I’ve wanted to blog about before but didn’t. So please pardon the disjointed nature of the following notes.
 

Since I’m writing this while on the plane to the Software Architecture Workshop in Switzerland, I’m reminded of a discussion I had with Arjen Poutsma at this same workshop last year. I think that the original topic had to do with Web Services, but it meandered around quite a bit. I think it started by me saying that services should not expose CRUD style operations. Arjen countered by mentioning that most user interfaces in line-of-business applications exposed the same model to the user by having them fill out data in grids. My retort to that was that while humans can get used to almost anything as long as its consistent, that doesn’t mean that it is a good solution. In the systems that I work on there is usually an HCI (human-computer interaction) person on the project who designs the UI, mostly around the tasks they perform. These tasks often corresponded very well to the coarse-grained messages we employed in terms of SOA. We finally agreed that the successor of SOA would be TOA (Task-Oriented Architecture) in its aggregation of client-side aspects to the already server-centric principles of SOA.
 

A different topic that came up in a recent meeting in one of the projects I’m consulting was how long users expect to wait for a response from the system. What made the HCI person rethink his design was my suggestion that certain algorithms could be deployed client-side and since their performance for the kinds of work the user would do most of the time was on the order of a couple of hundered millis, we could run them “interactively” – as in, on every mouse-click. This input eventually brought about a greatly simplified and much more interactive experience for these expert users. I’m no HCI expert, but I’ve learned a thing or two over the years, and one of the important ones was the need for synergy between architecture and HCI design.
 

Finally, code that supports highly interactive user interfaces is non-trivial to say the least. Jeremy brought this point up and suggested using design patterns like MVP with a healthy dose of unit tests. I couldn’t agree more, well, yes I could. Even if you use the next generation patterns of Passive View and Supervising Controller, the Dependency Injection development style, and the Command Object pattern, it will not be enough! I’m seeing, in real time, what happens to a project that utilizes all the appropriate patterns, manage their dependencies well, decouple fervently with events, and keep their code clean at all times but doesn’t require/encourage developers to write unit tests. It is a stability nightmare. If you have a complex system to build with intricate logic as to what can be activated when, or any long list of detailed requirements in terms of user interaction, ignore unit tests at your peril. Having a testable design is a great first step, but if you don’t go and test, you’ve negated quite a lot of its benefits.
 

I hope that that didn’t amount to just a bunch of over-tired, jet-lagged, incoherent babbling, but I’ve been waiting to get it off my chest and now seemed like just the time to do so.

More information:

Usability is Timeless, and things that are still broken with usability today.