Quick post on general “goings on”.

SOA & DDD Early Bird

Jeffery already announced the early bird discount for the course I’m giving in October. You get 10% off for the next week or so. Sounds good.

Grid Presentation

I’m going to be giving my Avoid a Failed SOA talk to the Israeli Association of Grid Technologies on September 2nd. Full details here.

97 Nuggets

I’m really honoured to have taken part in the 97 Things Every Software Architect Should Know project. I don’t think there’s ever been such a concentration of world-class architects on any one project (I really don’t know how I got past the bouncer). Take a look.

IASA Webinar

I’ve mentioned before that I’m working on a course on Reliability, Availability, and Scalability for the International Association of Software Architects (IASA). That’s going to take a bit longer to get out, so in the mean time I’m going to be giving a webinar September 25th on the back of the IT Architect Regional Conference in Philadelphia. I’ll post the registration link when it comes online.

SD Best Practices

Finally, the guys from Dr. Dobb’s are going to be working me to the bone at the end of October for SD Best Practices in Boston . On top of the full-day nServiceBus tutorial, I’m going to be speaking every day. You can find my full list of sessions here. Apparently today’s the last day for the $700 early bird discount too.

Come up and say hi if you’re at one of these events. It’s always great meeting my readers.

The good folks from HeadSpring are bringing me to Austin, TX this October to teach my Advanced Distributed Systems Design with SOA & DDD course.

You can find the syllabus online here but I’ll give you the short and sweet version:

What I’ve done is boiled down several months of consulting on everything dealing with distributed systems development into 5 jam-packed days of must-haves, need-to-knows, patterns, pitfalls, anti-patterns, rules-of-thumb, examples, and case studies.

Take a look.

To the question of scale Ayende brings up, I thought I’d tap my concept map.

First of all, I wanted to address the relationship between various topics related to scalability:

And on the connection between scalability and throughput:

The important message here is that the scalability of a system is a cost function that gives throughput as a function of recurring costs and one time costs – servers and other hardware, and the join of buy & build:

Did you write your own locking/transaction mechanism on top of an open source distributed cache or did you buy a license for a space-based technology?

Also, don’t forget that people need to administer all the servers that you have. Those people cost money (easily100K per year). Maybe, because you haven’t invested in management or monitoring tools you need one person for every two servers. This will influence the breakdown of up front costs and recurring costs. Also, the level of availability you require will impact this as well.

In my experience, architects don’t consider often enough the operations environment in their "scalability calculations".

What this means is that there’s no such thing as technically "not being able to scale".

Rather, that the cost (up front + recurring) of supporting higher throughput grows faster than the function of revenue per user/request/whatever.

Sometimes, the solution is just to find ways to make more money per customer.

For more technical solutions, take a look at the difference between capacity and scalability and how the competing consumer pattern helps scale out.

Scalability, it’s all about the money.

—

Oh, I almost forgot, I also had a great conversation with Carl and Richard about scaling web sites that’s now up on the .NET Rocks site. Enjoy.

One of the common questions I receive from people starting to use nServiceBus is how one-way messaging fits with showing the user a grid (or list) of data. Thinking about publish/subscribe usually just gets them even more confused. Trying to resolve all this with Service Oriented Architecture leaves them wondering – why bother?

In regular client-server development, the server is responsible for providing the client with all CRUD (create, read, update, and delete) capabilities. However, when users look at data they do not often require it to be up to date to the second (given that they often look at the same screen for several seconds to minutes at a time). As such, retrieving data from the same table as that being used for highly consistent transaction processing creates contention resulting in poor performance for all CRUD actions under higher load.

A Scalable Solution

One of the common answers to this question is for the server/service to publish a message when data changes (say, as the result of processing a message) and for clients to subscribe to these messages. When such a notification arrives at a client, the client would cache the data it needs. Then, when the user wants to see a grid of data, that data is already on the client. Of course, this solution doesn’t work so well for older client machines (like some point of service devices) or if there are millions of rows of data.

The thing is that this solution is one implementation of a more general pattern – command query separation (CQS).

Command Query Separation

Wikipedia describes CQS as a pattern where "… every method should either be a command that performs an action, or a query that returns data to the caller, but not both. More formally, methods should return a value only if they are referentially transparent and hence possess no side effects."

Martin Fowler is less strict about the use of CQS allowing for exceptions: "Popping a stack is a good example of a modifier that modifies state. Meyer correctly says that you can avoid having this method, but it is a useful idiom. So I prefer to follow this principle when I can, but I’m prepared to break it to get my pop."

So, how does separating commands from queries and SOA help at all in getting data to and from a UI? The answer is based on Pat Helland’s thinking as described in his article Data on the Inside vs. Data on the Outside.

Services Cross Boxes

The biggest lie around SOA is that services run.

Let that sink in a second.

Sure services have runnable components, but that’s not why they’re important.

I’ll skip the books of background and cut to the chase:

Services communicate with each other using publish/subscribe and one-way messaging. Services have components inside them. Inside a service, these components can communicate with each using synchronous RPC, or any other mechanism. Also, these components can reside on different machines.

This is broader than just scaling out a service. There can be service components running on the client as well as the server.

SOA & CQS

Combining these two concepts together, here’s what comes out:

In this solution there are two services that span both client and server – one in charge of commands (create, update, delete), the other in charge of queries (read). These services communicate only via messages – one cannot access the database of the other.

The command service publishes messages about changes to data, to which the query service subscribes. When the query service receives such notifications, it saves the data in its own data store which may well have a different schema (optimized for queries like a star schema).

The client component which is in charge of showing grids of data to the user behaves the same as it would in a regular layered/tiered architecture, using synchronous blocking request/response to get its data – SOA doesn’t change that.

Composite Applications

Although the client side components of both the command and query services are hosted in the same process, they are very much independent of each other. That being said, from an interoperability perspective (the one that most people attribute to SOA), all of the client-side components will likely be developed using the same technology – although there are already ways to host Java code in .NET and vice-versa.

Of course, once we talk about web UI’s things are a bit different – but still similar. While web-server-side there may be a level of independence, for browser side inter-component communications we’re still likely to target javascript. There, I’ve managed to say something technical supporting mashups and SOA without lying through my teeth.

On the Microsoft side with the recent release of the Composite Application Guidance & Library (pronounced "prism") I hope that more of these principles will be reaching the "smart client". The command pattern is especially critical in maintaining the separation while enabling communication to still occur so I’m glad that, as one of the Prism advisors, I was able to simplify that part (Glenn still has nightmares about that rooftop conversation).

Publish / Subscribe

In the "scalable solution" section up top I mentioned how publish/subscribe to the smart client is really just one implementation of CQS and SOA. So, how different is it really?

Well, there will probably be a different technology mapping. Instead of a star-schema OLAP product, we might simply store the published data in memory on the client. That is, if you designed your components to be technology agnostic.

In terms of the use of nServiceBus, the same component is going to be subscribing to the same type of message – all that’s different is that now every client will be having data pushed to them rather than this occurring server-side only.

You could have the same code deployed differently in the same system – stronger clients subscribing themselves, weaker ones using a remote server. Web servers would probably be considered stronger clients. This kind of flexible deployment has proven to be extremely valuable for my larger clients. The added benefit of enabling users to work (view data) even while offline (somewhere there’s no WIFI) is just icing on the cake.

A Word of Warning

Once the client starts receiving notifications, and handling those on a background thread (as it should) the code becomes susceptible to deadlocks and data races. Juval does a good job of outlining some of those with respect to the use of WCF. Prism doesn’t provide any assurances in this area either.

Summary

NServiceBus is not designed to be used for any and all types of communication in a given architecture. In the examples above, nServiceBus handles the publish/subscribe but leaves the synchronous RPC to existing solutions like WCF. Not only that, but synchronous RPC does have its place in architecture, just not across service boundaries. In all cases, data is served to users from a store different from that which transaction processing logic uses.

Command Query Separation is not only a good idea at the method/class level but has advantages at the SOA/System level as well – yet another good idea from 20 years ago that services build upon. Making use of CQS requires understanding your data and its uses – SOA builds on that by looking into data volatility and the freshness business requirements around it.

Finally, designing the components of your services in such a way that their dependency on technology is limited buys a lot of flexibility in terms of deployment and, consequently, significant performance and scalability gains.

Simple, it is. Easy, it is not.

The other day I had this idea, what if I were to take all the concepts I write, speak, and consult about and turn them into a concept map. That might help me explain how things like messaging, unit of work, and exception management work together and why. It also shouldn’t be too much work. Or so I thought.

I started out with a blank piece of paper, and this is what happened:

I got into some threat modeling, which connected to authentication, authorization, and integrity, connecting to consistency, transactions, and fault tolerance, and, on the flip side, eventual consistency, messaging, and REST. Yes, SOA is in a tiny circle at the bottom left 🙂

I tried to keep the file big enough to be quite readable but small enough to be sent directly via email (less than 600KB).

The coming release is going to be a more interactive environment where you’ll be able to click on each concept for more information, see what else its linked to, and get links to online resources. This is quite an undertaking so if this is something that you want to see move forward, please leave a comment or maybe link to it from your blog. It’s hard for me to know what really connects with you and what you just delete from your reader so any feedback is really appreciated.

Don’t.

Not in applicative code anyway.

This follows up on Ayende’s post about the AOP way.

Now, I have nothing against AOP but some developers are leery of it.

In broader terms, all logging goes in framework-level code. For smart clients, one really good place to put logging is in your Command infrastructure – every time a command is invoked, log it and the args. For data access, well, any decent O/R Mapper has a lot of logging already, just use it. For communication, ditto. Funny that just last week this was one of the major bits of feedback I gave in a code review.

The Important Part

Logging is useful for developers to find out why a system isn’t working correctly.

It is terrible for knowing that a system isn’t working correctly.

If you’re entire exception management strategy is “write it to the log”, how will an admin know that something’s wrong? Did you remember to configure your logging library that errors (and maybe warnings too) should be pushed out to a monitoring system? Do you have a monitoring system?

And if the admins don’t know anything’s wrong, they won’t know they need to increase the fidelity of the logs, will they? Are you planning on providing training for your admins telling them this (and all the other things they need to know)? Or maybe this will all be set up as an automatic script?

An Agile Digression

I hope all that’s on your agile (“we can ship at the end of every 2 week iteration”) product backlog (pardon my cynicism). I hope it’s at least something that you’re looking at per release and feeding the relevant features into your iterations. Yes, there’s project work to do (writing training manuals) that isn’t “development” that needs to be handled; if you don’t timebox it into the same iterations, it won’t get done.

Now, back to you’re regularly schedule logging…

Things Logging Doesn’t Address

Logging is a mildly useless tool for pinpointing where in the system the source of a problem is.

“I know the entity isn’t in the database. I can see that. I want to know why it isn’t there.”

Sure, if you had every SQL statement logged you could figure these sorts of things out out. Of course, performance-wise, you wouldn’t put the system into production like that. In which case, the delete statement wouldn’t have been logged leaving you with precious little information to solve the root cause.

Also consider that the more logging you do, the more crap you’ll have to sift through to find the proverbial needle. Developers often don’t think twice about increasing the amount of crap logs they generate…

The Real Problem

The real problem is that developers think too much about logging and not at all nearly enough about designing the system in ways that it’ll be easy possible to answer questions like those above without having to know exactly how the system is built. One of the reasons that developers should care about this is that it’ll decrease the number of times they need to get up at 3:00 am to answer those questions.

A Path to the Solution

Now, if you had some kind of business activity monitoring (BAM) capability in your system, an admin could do a simple search/query [WHEN entity DELETED] and find out answers to the questions above, find out the time that the relevant activities occurred, figure out what the problem is on their own, and maybe even fix it – especially if it has to do with some esoteric configuration variable.Regardless of whether you buy a BAM tool or roll what you need yourself, you need to understand what about the system needs to be monitored. That’s a very different thought-process to go through than “should we log this? Yeah, sure, why not.”

It’s called “Design for Operations”.

Take a holistic perspective on exception management, logging, monitoring, etc. Think about questions like those above and then analyse your use of the relevant tools in that context. Think about all the different kinds of users of the information that’s going to be generated and how quickly their going to need to act on that information. Admins in the data-center in the middle of a crisis are going to have different needs than developers analysing logs on their machine. Think about:

• How will the administrator know that a server has been configured properly?
• If the system is feeling slow, how can the administrator know which server/process is to blame?
• So that maybe they can scale out that part of the system.

In Closing

It’s a mindset.

It takes time to make the shift.

It takes more time to bring the development process to this kind of maturity (god, I hate that word).

Writing exceptions to the log is not a strategy.

At the very best, its a tactic.

What’s your strategy?

While I was at TechEd USA I had an attendee, Will, come up and ask me an interesting question about how to handle web service calls that can take a long time to complete. He has a number of these kinds of requests ranging from computationally intensive tasks to those requiring sifting through large amounts of data. What Will was having problems with was preventing too many of these resource-intensive tasks from running concurrently (causing increased memory usage, paging, and eventually the server becoming unavailable).

For comparison later, here’s a diagram showing the trivial interaction:

One solution that he’d tried was to set up the web server to throttle those requests and keep a much smaller maximum thread-pool size for that application pool. The unfortunate side effect of that solution was that clients would get “turned away” by a not-so-pleasant Connection Refused exception.

Will had been to my web scalability talk and was curious about how I was using queues behind my web services. I’ve also heard this question from people just getting started with nServiceBus when looking at the Web Services Bridge sample. Here’s the code that’s in the sample and in just a second I’ll tell you why you shouldn’t do this:

[WebMethod]

public ErrorCodes Process(Command request)

{

    object result = ErrorCodes.None;

    IAsyncResult sync = Global.Bus.Send(request).Register(

        delegate(IAsyncResult asyncResult)

          {

              CompletionResult completionResult = asyncResult.AsyncState as CompletionResult;

              if (completionResult != null)

              {

                  result = (ErrorCodes) completionResult.ErrorCode;

              }

          },

          null

          );

    sync.AsyncWaitHandle.WaitOne();

    return (ErrorCodes)result;

}

Let me repeat, this is demo-ware. Do not use this in production.

What’s happening is that in this web service call we’re putting a message in a queue for some other process/machine to process. When that processing is complete, we’ll get a message back in our local queue (which you don’t see) which is correlated to our original request, firing off the callback. We block the web method from completing (using the WaitOne call) thus keeping the HTTP connection to the client open.

The problem here is that we’re wasting resources (the HTTP connection and the thread) while waiting for a response which, as already mentioned, can take a long time. In B2B or other server to server integration environments there are all sorts of middleware solutions that help us solve these problems, however in Will’s case browsers needed to interact with this web service. All he had was HTTP.

HTTP Solutions

Another attendee who was listening in (sorry I don’t remember your name) said that he was solving similar problems using polling but that he was having scalability problems as well.

What often surprises my clients when we deal with these same issues is that I do suggest a polling based solution, but one that still uses messaging, and this is what I described to Will:

Since we can’t actually push a message to a browser over HTTP from our server when processing is complete, the browser itself will be responsible for pulling the response. We still don’t want to leave costly resources like HTTP connections open a long time, however if the browser is going to polling for a response, we’ll need some way to correlate those following requests with the original one. What we’re going to do is use the Asynchronous Completion Token pattern, and later I’ll show how to optimize it for web server technology.

Basic Polling

When the browser calls the web service, the web service will generate a Guid, put it in the message that it sends for processing, and return that guid to the browser. When the processing of the message is complete, the result will be written to some kind of database, indexed by that guid. The browser will periodically call another web method, passing in the guid it previously received as a parameter. That web method will check the database for a response using the guid, returning null if no response is there. If the browser receives a null response, it will “sleep” a bit and then retry.

One of the problems with this solution is that polling uses up server resources – both on the web server and our DB; threads, memory, DB connections. A better solution would decrease the resource cost of the polling. Let’s use the fundamental building blocks of the web to our advantage – HTTP GET and resources:

REST-full Polling

Instead of using a guid to represent the id of the response, let’s consider the REST principle of “everything’s a resource”. That would mean that the response itself would be a resource. And since every resource has a URI, we might as well use that URI in lieu of the guid. So, instead of our web service returning a guid, let’s return a URI – something like:

http://www.acme.com/responses/88ec5359-a5d8-4491-a570-3bfe469f3a64.xml

As you can see, the guid is still there. So, what’s different?

What’s different is that instead of having the processing code write the response to the database, it writes it to a resource. This can be done by writing some XML to a file on the SAN in the case of a webfarm. Also, the browser wouldn’t need to call a web service to get the response, it would just do an HTTP GET on the URI. If the it gets an HTTP 404, it would sleep and retry as before. The reason that the SAN is needed is that, as the browser polls, it may have its requests arrive at various web servers so the response needs to be accessible from any one of them.

Just as an aside, it would be better to free the processing node as quickly as possible and have something else write the response to the SAN. That would be done simply by sending a message from the processing node that would be handled by a different node that all it did was write responses to disk.

The reason that the URI makes a difference is that serving “static” resources is something that web servers do extremely efficiently without requiring any managed resources (like ASP.NET threads). That’s a big deal.

We’re still using HTTP connections for the polling but that’s something whose effect can be mitigated to a certain degree.

Timed REST-full Polling

Since various requests can take varying amounts of time to process, it’s difficult to know at what rate the browser should poll. So, why don’t we have the web service tell it. As a part of the response to the original web service call, instead of just returning a URI, we could also return the polling interval – 1 second, 5 seconds, whatever is appropriate for the type of request. This value could easily be configurable [RequestType, PollingInterval].

An even more advanced solution would allow you to change these values dynamically. The advantage that would be gained would be that your operations team could better manage the load on your servers. When a large number of users are hitting your system, you could decrease the rate at which your servers would be polled, thus leaving more HTTP connections for other users.

Scaling and Adaptive Polling

You’d probably also want to scale out the number of processing nodes behind your queue. The nice thing is that you could change the polling interval as you scale the various processing nodes per request type providing better responsiveness for the more critical requests. Once we add virtualization, things get really fun:

We had separate queues per request type, so that we could easily see the load we were under for each type of request. That way, we could scale out the processing nodes per request type as well as change the polling interval. By virtualizing our processing nodes, and writing scripts to monitor queue sizes, we had those scripts automatically provisioning (and de-provisioning) nodes as well as changing the polling interval of the browsers.

This had the enormous benefit of the system automatically shifting resources to provide the appropriate relative allocation for the current load as its macroscopic make-up changed.

Summary

Will was well-pleased with the solution which, although more complicated than what he had originally tried, was flexible enough to meet his needs. As opposed to pure server-based solutions, here we make more use of the browser (writing our own Javascript) instead of putting our faith in some Ajax-y library. That’s not to say that you couldn’t wrap this up into a library – in essence, it is a kind of messaging transport for browser to server communication allowing duplex conversations.

In fact, what could be done is to return multiple responses to the browser over a long period of time. In the response that comes back to the browser could be an additional URI where the next response will be. This can be used for reporting the status of a long running process, paging results, and in many other scenarios.

And, one parting thought, could this not be used for all browser to web service communication?

And louder than ever. That’s right, I’m still an MVP.

You can find the PDF of the presentation I gave at Dr. Dobb’s Architecture & Design World 2008 online here.

Enjoy.

I’ve received some great feedback on my MSDN article and some really great questions that I think more people are wondering about, so I think I’ll try to do a post per question and see how that goes.

Libor asks:

“Would you recommend using durable messaging for systems where there are similar requirements with respect to data reliability as you had – ie. not losing any messages? If so, then why didn’t the final version of your solution use it? If not, can you explain why?”

The answer is, as always, it depends, but here’s on what it depends:

When designing a system, we need to take a good, hard look at how we manage state, and what properties that state has. In a system of reasonable size we can expect various families of state with respect to their business value, data volatility, and fault-tolerance window. Each family needs to be treated differently. While durable messaging may be suitable for one, it may be overkill or underkill for another.

So, here’s what we’re going to be looking at:

1. Business Value
2. Data Volatility
3. Fault-Tolerance Window

Business Value

When talking about business value, I want to talk about what it means “not losing any messages”. The question is under what conditions will the messages not be lost, or rather, what are the threshold conditions where messages may start getting lost. If all our datacenters are nuked, we will lose data. It’s likely the business is OK with that (as much as can be expected under those circumstances). If a single server goes down, it’s likely the business would not be OK with losing messages containing financial data. However if a message requesting the health of a server were to get lost under those same conditions, that would probably be alright. In other words, what does that message represent in business terms.

Data Volatility

Data volatility also has an impact. Let’s say that we’re building a financial trading system. The time that it takes us to respond to an event (message) that the cost of a certain financial instrument has changed, and the message that we send requesting to buy that security is critical. Let’s say that has to be done in under 10ms. Now, some failure has occurred preventing our message from reaching its destination for 20ms. What should we do with that message? Should we keep it around, making sure it doesn’t get lost? Not in this domain. On the contrary, that message should be thrown away as its “business lifetime” has been exceeded. Furthermore, even during that original period of 10ms, the use of durable messaging may make it close to impossible to maintain our response times.

Fault-Tolerance Window

These two topics feed into the third and more architectural one – fault-tolerance window: what period of time do we require fault tolerance, and with respect to how many (and what kind of) faults? This will lead us into an analysis of to how many machines do we need to copy a message before we release the calling thread. We’d also look at in which datacenters those machines reside. This will also impact (or be impacted by) the kinds of links we have to these datacenters if we want to maintain response times. These numbers will need to change when the system identifies a disaster – degrading itself to a lower level of fault-tolerance after a hurricane knocks out a datacenter, and returning to normal once it comes back up.

Re-Evaluating Durable Messaging

Durable messaging may be used at various points in each part of the solution, but we need to look at message size, the rate those messages are being written to disk, how fast the disk is, how much available disk we have (so we don’t make things worse in the case of degraded service), etc. Companies like Amazon also take into account disk failure rates, replacement rates (disks aren’t replaced immediately you know), and many other factors when making these decisions

Summary

Our job as architects when designing the system is to find that cost-benefit balance for the various parts of the system according to these very applicative parameters. No, it’s not easy. No, cloud computing will not magically solve all of this for us. But, we are getting more technical tools to work with, operations staff is getting better at working with us in the design phase, and our thought processes more rigorous in dealing with the scary conditions of the real world.

To your question, Libor, as to why we didn’t eventually use durable messaging in our solution, the answer is that we solved the overall state management problem by setting up an applicative protocol with our partners which was resilient in the face of faults by using idempotent messages that could be resent as many times as necessary. You can read more about it here. This solution isn’t viable for other kinds of interactions but was just what we needed to get the job done.

Hope that helps.