I’ve been getting some questions from the Dependency Injection folks out there as to why I have my own Object Builder wrapping the framework. There are two very good reasons why I do this:

The first is to insulate the framework and application code that I write from the choice of one dependency injection technology or another. I want the ability to switch easily from one to the other – not so much that projects go back and forth. Updating those config files is definitely not easy. However, it allows me to have “portable” framework code that is applicable to all the projects I consult on, regardless of their choice of technology.

The second has to do with NServiceBus specifically. In order to make use of duplex communication on smart clients, you need a background thread. That thread will be updating the same (model) objects as the UI thread. That means we need synchronization. I prefer to use .NET’s built-in synchronization domains in order to solve this rather thorny problem.

The only thing is that message handlers need to be in the synchronization domain so that they can easily update those objects. However, the Bus object must not be in the synchronization domain so that if we’ve received a large update from the server, we won’t be locking out the UI thread from interacting with data on the client.

Since the bus makes use of a dependency injection framework to create message handlers, this was the best place to put the code which causes message handlers to run within the synchronization domain.

Be aware that in order to enjoy this feature, you need to split up those large server updates into multiple, logical objects (that implement IMessage), but you can still publish them all in one go using the method:

void Publish(params IMessage[] messages);

And, of course, you need to set the JoinSynchronizationDomain property of the Object Builder.

I’ll have a podcast coming out on this topic soon.

You can get the code here:

Object Builder.zip

But you’ll have to get the Spring Framework code from the official site. Make sure you download RC 1.1. Then, take the binaries and copy them to the “BIN” folder of the Object Builder solution. If you’re looking to save on some “weight”, you only need “Spring.Core.dll”, “Common.Logging.dll” and “antlr.runtime.dll” for the solution to compile. You will need one of the logging implementations DLLs to get anything written to a log, obviously.

This week we discuss the different options that are available for passing data between a client and a service, as well as common pitfalls around trying to use the same objects for both presentation to the user and persistence to the database.

Download via the Dr. Dobb’s site.

Or download directly here.

And here’s the question that gets us started:

Hello there,

We are developing an application in .NET Smart Client application which connects to a WCF service and then to the data access layer. The issue we have here is on the type of data which can be passed between the client side and web service.

1. Can it be a dataset (No!), custom entity or XML or anything else ?

Which is the best one ?

2. Ofcourse , a dataset can’t be used coz it can’t be used in non-.NET environment?

3. If we go for custom-entity or XML how do we ensure the changes happen to the data during client-side actions ? For example, binding the XML or custom entity to a data grid can be done. But if user makes any changes to the same, how do we identify ?

4. If we use custom entities / XM L, the same can not be passed to the Data Access Application Block which invokes the data objects.

Note : All the above are based on assumption without using a translator in between any of the layers.

Please provide a better solution if you have come across any.

Additional References:

• Podcast on using DataSets with Web Services
• Blog post on tracking changes to objects on the client side
• Looking for more of the “Ask Udi” podcast?

After reading Nick Malik’s recent post describing what he likes about Acropolis I was reminded of a conversation I had with the guys behind the CAB. Nick writes:

What I find promising: an Acropolis part can essentially consume a SOA service, allowing the composition of process and activity services to be as simple as snapping parts onto a surface.

Since services which conform to SOA principles will expose duplex/asynchronous APIs, this will almost necessarily make the client-side interaction multi-threaded. I do, and have done quite a lot of work on multi-threaded, thread-safe, smart clients. They’re hard. Here’s a post describing how hard. There are other solutions as well, but very little of it has to do with technology.

Since both the CAB and Acropolis do very little in handling, or even describing in their docs these multi-threading issues, I don’t think that they’re going to make it simple or easy to work with SOA-style services. This is not to say they don’t provide any value. On the contrary, I think that they bring the mainstream many steps forward. However, there is still a long way to go.

I’ve gotten a couple of questions on my MSMQ implementation of an ESB as to why I chose to take the definition of the CompletionCallback and put it in its own project, rather than just putting it together with the IBus interface.

The answer to this is that this callback is designed for Controller classes in Smart Clients, at least when employing an MVC style. You see, as the Controller class is the one who decided to initiate the action on the remote service by calling into a Service Agent class, it will also contain the logic for what to do when a result arrives. Therefore, it needs to pass a callback to the Service Agent class, possibly also passing in some other state (for instance, a reference to the form for which it is making this call, if there can be multiple instances of the same type of form open).

The only dependency the Controller class has is on the fact that there is asynchronous communication happening – made clear by the tiny assembly containing only the callback definition. The Controller does not need to know anything about ESBs and such, and thus does not need a reference to the IBus interface project.

As always, when working asynchronously you need to be aware that these callbacks will be coming in on a thread other than the UI thread. This means that if your Controller class has any state (ie. member variables), you need to protect that.

Ayende brought up the topic of Input & Business Rule Validation and I wanted to post how I solve this issue.

On kind of input validation is something you do as close to the user as possible for performance reasons. This includes all sorts of smart stuff you can do with JavaScript in Web scenarios. When in a Smart Client environment, you usually have greater capabilities.

When I look at the issue of validation, I see that it centers around the entity. Sometimes, it is also affected by other things, like what process are we in (as described in the comments on Ayende’s post).

So, we can model the thing that validates an entity with an interface, say, IValidator<T> where T : IEntity. This interface will have one main method: bool IsValid(T entity); and one main property: string ErrorDescription { get; }

What this allows us to do is to separate out different validation concerns into different classes, yet have all of them implement the same interface.

The next thing we’ll need is to be able to get an instance for each of the classes that is a validator for a specific kind of entity. For instance, when the NewCustomerView raises an event proclaiming that it has a Customer object ready to be saved, the Controller will want to find all classes that implement IValidator<Customer> so that it can run all the validation rules.

Luckily, the generics patch I put out for the Spring.Net Framework allows us to do this in one simple line of code:

IList<IValidator<Customer>> validators = spring.GetObjectsOfType(typeof(IValidator<Customer>));

and quite simply perform the validation as follows:

foreach(IValidator<Customer> v in validators)
  if (!v.IsValid(myCustomer))
    // notify user with v.ErrorDescription, write to log, whatever

Now, when using this in Smart Client scenarios, you will often have views that allow the user to enter in a single entity which you will then want to validate. If you have those views implement a generic interface like: IEntityView<T> where T : IEntity, then you could have a single base class implement a “Validate” method, which would perform the work above generically like so:

foreach(IValidator<T> v in spring.GetObjectsOfType(typeof(IValidator<T>)))
  if (!v.IsValid(this.Entity))
    // notify user with v.ErrorDescription

and just have your specific view call that method on the button click.

This enables all entity views to activate all the relevant custom validation logic without being tied to it. It also enables you to extend your system by adding new classes implementing the IValidator<T> interface, and have them automatically run without even changing a config file. How’s that for loose coupling?

Finally, on the issue of tying validation rules to specific processes, this can be done by extending the interface to: IValidator<T, P> where T : IEntity, P : IProcess; You then model each process with a marker interface. You then have your specific validation classes implement the above generic interface for each specific process interface. For instance, say we have a validation rule that needs to run for processes P1 (marked by IP1), and P2 (marked by IP2), but not P3 (marked by IP3), which validates entities of type E. This would be done by defining the class like so:

public class MyValidator : IValidator<E, IP1>, IValidator<E, IP2> {}

and the Controller class that would request validators for process 2 would just call:

spring.GetObjectsOfType(typeof(IValidator<T, IP2>));

That’s it. The basic principles are simple, but, as you can see, can create very powerful structures. I’ve got to say that this exemplifies one of the reasons why I love generics so much. When used with Dependency Injection, and/or Delegates, and/or Anonymous Methods, you get such a power of expression just by defining an interface. This is one of things that make coding fun for me. Or maybe I’m just wierd that way 🙂

“How come there is no talk about scaling these ORMs?”.

The answer is simple.

You don’t have to.

Or, if you think you do, you’re probably using them wrong.

An O/R mapper should NOT live in its own tier.

Object/Relational Mappers never stand alone. They are used to provide persistence for something else – either for the Domain Model Pattern or for the Active Record pattern.

In terms of scalability, again these patterns usually don’t stand by themselves, but rather are hosted by something else. When used on the client side, say in a smart client application, then scalability isn’t often considered. There are some kinds of smart clients where hitting the database on most user interactions will bring the system to its knees, but that’s again an issue of database scalability. The common solution is some kind of client-side caching.

Anyway, the two main parameters you need to look at for your common and high-load scenarios when using an O/R mapper are these:

• How many times do you hit the DB per business action.
• How many objects/records/rows/columns do you bring into memory per business action.

You should be looking at appropriate uses of Lazy Loading for both of them.

Finally, keep in mind that O/R mappers are only part of your solution. Measure and optimize wisely.

This week’s question comes from Oran who asks:

Hi Udi,

I’m enjoying the recent discussion on Entity Framework, disconnected clients, Unit of Work, and messaging. A few weeks ago I wrote a note to self to “Ask Udi” about the new ADO.NET Sync Services coming out of Microsoft, and this seems to be the perfect timing.

It really seems that Sync Services is meant to address the disconnected problem raised by Andres, but it appears to be primarily targeted at enabling DataSet-centric, grid-oriented UIs, not task-oriented UIs. They are spinning Sync Services as being an “SOA” way of doing data synchronization because “it uses WCF” and ” supports n-tier”. But under the hood it’s all about DataSet-style deltas. See Steve Lasker’s first Q&A for the SOA spin, and the rest of his blog for more info.

I’m curious what your thoughts are on the subject. Do you see Sync Services as a valid service-oriented solution for occasionally-connected clients, or is it just another attempt by the ADO.NET team to keep us all hooked on DataSet-driven development?

Oran

Get it via the Dr. Dobb’s site here.

Or download directly here.

Additional References

• Occasionally Connected Systems Architecture
• Entity Framework: Disconnected Problems & Solutions
• Tasks, Messages, & Transactions–the holy trinity

Want more? Go to the “Ask Udi” archives.

Many systems don’t operate in environments where “client” machines have uninterrupted connectivity with “server” machines. This has already been stated more generally in the “8 Fallacies of Distributed Computing” – the network is not reliable. Let’s take a look at some patterns that alleviate many of the burdens of handling these difficult scenarios. We’ll be going through these patterns broken down primarily by the list that Roger Jennings published about some of the architectural characteristics of Occasionally Connected Systems (OCS). I’ll be addressing the points Frans raised in his in-depth analysis of Entity Change Tracking in a separate post.

The lowly GUID

One of the simpler patterns that, when viable, keeps basic things simple is that of Client-side Identifier Generation –  if client-side code is capable of creating entities and giving them an identifier which will not be changed at the time those entities are persisted. The simplest implementation of this pattern is to use a Globally Unique Identifier (GUID) to identify entities in the systems. By having client-side code creating objects whose identities will be correct even before saving them, we are saved the chore of refreshing all the objects the client created when it eventually gets connected to the server again and can save those objects.

One requirement that Roger mentions moves the architecture more towards an “N-Tier” style and further from the classic “Client-Server” style:

“Locally cached lookup (catalog or historical) data, which can be very large and thus must merge changes rather than require full-table refreshes”

Messaging & Pub/Sub

By employing the publish/subscribe pattern between clients and an application server, as well as durable subscriptions when clients get connected they get pushed only the updates since the last time they were connected. This pattern improves scalability in that it prevents the scenario of many client connecting and performing costly queries against the database at the same time. Another difference between this (messaging-based) pattern and the more common database-level synchronization is that it maintains the discrete units-of-work (pattern) that were performed. While this isn’t very important for updates being pushed from the server to the client, it is absolutely critical for changes the client wants to send the server. These are the relevant architectural characteristics:

·         “Automatic or semi-automatic push of cached data to the server (two-tier) or data service (n-tier) when connected to the network”

·         “A process to enable the client to resolve data update and deletion concurrency conflicts; alternatively, to notify the client of the action taken by a business rule”

·         “A process to enable the client to resolve insert and other conflicts with multiple rows of child tables, such as order or medication line items”

Be neither overly optimistic nor overly pessimistic

One of the greatest challenges of the business logic in these systems is the handling of concurrency conflicts – multiple users changing the same set of data independent of each other and now trying to “upload” those changes to the server. This is handled on the server-side by the techniques I detailed in “Realistic Concurrency”, but it is critical to have the client pass the Units-of-Work it performed to the server for that to work. By modelling these units-of-work as messages that the client sends to the server, buffering them when there’s no connectivity, we are able to employ these Connected-Systems patterns as well.

Caching

Another interesting question is how the data is cached on the client. Before we get into the solutions, I think that we need to divide this question into two parts. Given a connected system, what is the best way to structure data on the client side; and does the fact that the client is occasionally connected change anything? Ayende suggest separate data structures for the client and server. I strongly agree – regardless of connectivity. I suppose that this is one of the reasons that I don’t particularly like using datasets on the client – it is so strongly tied to persistence concerns that don’t worry me on the client.

A different model for the client

As for the second question – does connectivity change the client-side model? I have found that it does, but that it’s much more an issue of the kind of work that can be performed disconnected. If we choose to employ asynchronous messaging, we may need to handle cases where, due to message-design considerations, the client can get published child objects before parents. It is fairly straight-forward to handle this by simply storing both the parent ID and a reference to the parent object in child objects. In order to keep this data correct, it wise to update objects only by calling business-level methods on them rather then calling multiple setter-methods one after another.

One other issue is where these objects are stored on the client-side. In connected scenarios, this issue doesn’t exist until caching is involved. What I often use is a derivative of the Repository pattern. This repository enables searching for objects on the client-side, persists itself to some durable storage (like the file system) when the client application is shut down, and loads those objects from the storage on startup.

Wrapping up

While this isn’t an exhaustive treatment of all issues related to Occasionally Connected Systems Architecture, I hope that it serves as a good baseline for discussing techniques and technologies.

Related content

Podcast on Occasionally Connected Smart Clients
Podcast on passing data between layers in occasionally connected systems

Jesse is bringing more context to the previous discussion around developing occasionally connected clients (OCC) by adding SOA to the mix. In his latest post, he points out that the messaging based solution I presented does not necessarily make the system Service-Oriented. Jesse states:

“In fact, without a proper service oriented architecture and messages that are constructed based on tasks instead of CRUD operations, you will end up in the same boat as if you hadn’t implemented this approach.”

Let me be the first to agree.

In no way did I intend to discuss SOA related issues in my previous posts. Since I talk so much about SOA, and the use of the Bus Pattern and Asynchronous Messaging, I wanted to try to keep the discussion on OCC.

The main thing that I liked about Jesse’s response was that he agreed that my solution “solves some transaction issues” which is what I was aiming for. I’m going to be doing a separate series of posts on message design for SOA later, and would be happy to hear of anyone’s experiences in that area.

Andres Aguiar points out an enormous change to the behavior of data access in tiered architectures that is coming with the ADO.NET Orcas Entity Framework. He sums it up nicely, “Basically, there is no disconnected mode.” He then goes on to talk about the ramifications of this:

“This basically means that if you plan to build a multi-tier application with ADO.NET Orcas in the middle tier, you will need to hack your own change tracking mechanism in the client, send the whole changeset, and apply it in the middle tier. From this point of view, it’s a huge step backwards, as that’s something we already have with DataSets today.”

I do quite a bit of work on large-scale distributed systems, some that could even be called “multi-tier”, but I really don’t like that term. I’ve got to tell you, I’m not that worried. I could say that it’s because I’m using an object/relational mapping tool, but then again, the Entity Framework is supposed to be that too. It also might be because I don’t use DataSets that much today anyway, but that begs Andres’ question – did I develop my own change-tracking mechanism on the client?

The answer lies in the second part of Andres’ question, I don’t “send the whole changeset, and apply it in the middle tier”. I have an entire podcast up on why I don’t think that that’s a good idea, specifically around Web Services and SOA, but it’s broadly applicable. However, the question remains, what do we do about the client side?

Just over a year ago, me and Arjen Poutsma (of Spring fame) had a discussion about this exact topic at the Software Architecture Workshop in beautiful Cortina, Italy. Here’s the important part:

I think it started by me saying that services should not expose CRUD style operations. Arjen countered by mentioning that most user interfaces in line-of-business applications exposed the same model to the user by having them fill out data in grids. My retort to that was that while humans can get used to almost anything as long as it’s consistent, that doesn’t mean that it is a good solution. In the systems that I work on there is usually an HCI (human-computer interaction) person on the project who designs the UI, mostly around the tasks they perform. These tasks often corresponded very well to the coarse-grained messages we employed in terms of SOA. We finally agreed that the successor of SOA would be TOA (Task-Oriented Architecture) in its aggregation of client-side aspects to the already server-centric principles of SOA.

As a concrete example, instead of updating customer data in a grid showing everything laid out flat, users would activate tasks like “Change address”, or “Update marital status”. The form shown would contain only the relevant data to be updated – no need to perform “change tracking”, it is perfectly clear what data needs to be sent. Furthermore, the intent behind the changed data would be sent as well – the server wouldn’t apply the data generically “in the middle tier”. Rather, the current customer data would be retrieved as a part of the customer object, and the appropriate method called upon it (ChangeAddress, or UpdateMaritalStatus) with the relevant data.

I actually view this change in the Entity Framework as a step forward, it’s like taking off the training wheels. I’m positive that most developers will be able to make the change and move to these more advanced architectures.

This article has also been translated to Serbo-Croatian by Anja Skrba.