Udi Dahan   Udi Dahan – The Software Simplist
Enterprise Development Expert & SOA Specialist
    Blog Consulting Training Articles Speaking About

Archive for the ‘Threading’ Category

Domain Events – Take 2

Monday, August 25th, 2008

Update: The next post in this series is now online here.

My previous post on how to create fully encapsulated domain models introduced the concept of events as a core pattern of communication from the domain back to the service layer. In that post, I put up enough code to get the idea across but didn’t address issues like memory leaks and multi-threading. This post will show the solution to those two critical points.

I’ve snipped out one of the events in the previous example for brevity.

Previous API

The previous API looked like this:

   1:  public static class DomainEvents
   2:  {
   3:       public static event EventHandler GameReportedLost;
   4:       public static void RaiseGameReportedLostEvent()
   5:       {
   6:             if (GameReportedLost != null)
   7:                 GameReportedLost(null, null);
   8:       }
  10:       public static event EventHandler CartIsFull;
  11:       public static void RaiseCartIsFull()
  12:       {
  13:             if (CartIsFull != null)
  14:                 CartIsFull(null, null);
  15:       }
  16:  }

One thing that we want to keep in the solution is that all the code to define events, their names, and the parameters they bring will be in one place – in this case, the DomainEvents class. One thing that we’d like to fix is the amount of code needed to define an event.

Previous Service Layer

Here’s what our previous service layer code looked like:

   1:  public class AddGameToCartMessageHandler :
   2:      BaseMessageHandler<AddGameToCartMessage>
   3:  {
   4:      public override void Handle(AddGameToCartMessage m)
   5:      {
   6:          using (ISession session = SessionFactory.OpenSession())
   7:          using (ITransaction tx = session.BeginTransaction())
   8:          {
   9:              ICart cart = session.Get<ICart>(m.CartId);
  10:              IGame g = session.Get<IGame>(m.GameId);
  12:              Domain.DomainEvents.GameReportedLost +=
  13:                gameReportedLost;
  14:              Domain.DomainEvents.CartIsFull +=
  15:                cartIsFull;
  17:              cart.Add(g);
  19:              Domain.DomainEvents.GameReportedLost -=
  20:                gameReportedLost;
  21:              Domain.DomainEvents.CartIsFull -=
  22:                cartIsFull;
  24:              tx.Commit();
  25:          }
  26:      }
  28:      private EventHandler gameReportedLost = delegate { 
  29:            Bus.Return((int)ErrorCodes.GameReportedLost);
  30:          };
  32:      private EventHandler cartIsFull = delegate { 
  33:            Bus.Return((int)ErrorCodes.CartIsFull);
  34:          };
  35:      }
  36:  }

Another thing that should be improved is the amount of code needed in the service layer.

Raising an event, though, should still be fairly simple – one line of code similar to DomainEvents.RaiseGameReportedLost().


Here’s what the new API looks like:

   1:  public static class DomainEvents
   2:  {
   3:       public static readonly DomainEvent<IGame> GameReportedLost = 
   4:                                            new DomainEvent<IGame>;
   6:       public static readonly DomainEvent<ICart> CartIsFull=
   7:                                            new DomainEvent<ICart>;
   8:  }

It looks like we’ve managed to bring down the complexity of defining an event.

Raising an event is slightly different, but still only one line of code (“this” refers to the Cart class that is calling this API): DomainEvents.CartIsFull.Raise(this);

New Service Layer

The advantage of having a disposable domain event allows us to use the “using” construct for cleanup.

   1:  public class AddGameToCartMessageHandler :
   2:      BaseMessageHandler<AddGameToCartMessage>
   3:  {
   4:      public override void Handle(AddGameToCartMessage m)
   5:      {
   6:          using (ISession session = SessionFactory.OpenSession())
   7:          using (ITransaction tx = session.BeginTransaction())
   8:          using (DomainEvents.GameReportedLost.Register(gameReportedLost))
   9:          using (DomainEvents.CartIsFull.Register(cartIsFull))
  10:          {
  11:              ICart cart = session.Get<ICart>(m.CartId);
  12:              IGame g = session.Get<IGame>(m.GameId);
  14:              cart.Add(g);
  16:              tx.Commit();
  17:          }
  18:      }
  20:      private Action<IGame> gameReportedLost = delegate { 
  21:            Bus.Return((int)ErrorCodes.GameReportedLost);
  22:          };
  24:      private Action<ICart> cartIsFull = delegate { 
  25:            Bus.Return((int)ErrorCodes.CartIsFull);
  26:          };
  27:      }
  28:  }

I also want to mention that you don’t necessarily have to have the same service layer object handle these events as that which calls the domain objects. In other words, we can have singleton objects handling these events for things like sending emails, notifying external systems, and auditing.

The Infrastructure

The infrastructure that makes all this possible (in a thread-safe way) is quite simple and made up of two parts, the DomainEvent that we saw being used above, and the DomainEventRegistrationRemover which handles the disposing:

   1:  using System;
   2:  using System.Collections.Generic;
   4:  namespace DomainEventInfrastructure
   5:  {
   6:      public class DomainEvent<E> 
   7:      {
   8:          [ThreadStatic] 
   9:          private static List<Action<E>> _actions; 
  11:          protected List<Action<E>> actions 
  12:          {
  13:              get { 
  14:                  if (_actions == null) 
  15:                      _actions = new List<Action<E>>(); 
  17:                  return _actions; 
  18:              }
  19:          }
  21:          public IDisposable Register(Action<E> callback) 
  22:          {
  23:              actions.Add(callback);
  24:              return new DomainEventRegistrationRemover(delegate
  25:                  {
  26:                      actions.Remove(callback);
  27:                  }
  28:              ); 
  29:          }
  31:          public void Raise(E args) 
  32:          {
  33:              foreach (Action<E> action in actions) 
  34:                  action.Invoke(args);
  35:          }
  36:      }
  37:  }

Note that the invocation list of the domain event is thread static, meaning that each thread gets its own copy – even though they’re all working with the same instance of the domain event.

Here’s the DomainEventRegistrationRemover – even simpler:

   1:  using System;
   3:  namespace DomainEventInfrastructure
   4:  {
   5:      public class DomainEventRegistrationRemover : IDisposable 
   6:      {
   7:          private readonly Action CallOnDispose;
   9:          public DomainEventRegistrationRemover(Action ToCall) 
  10:          {
  11:              this.CallOnDispose = ToCall; 
  12:          }
  14:          public void Dispose() 
  15:          {
  16:              this.CallOnDispose.DynamicInvoke();
  17:          }
  18:      }
  19:  }

For your convenience, I’ve made these available for download here.

I also want to add that if you haven’t looked at the comments on the original post – there’s some really good stuff there (36 comments so far). Take a look.

Prism – Occasionally Connected?

Monday, June 9th, 2008

Prism, AKA Composite Application Guidance + Composite Application Library, is rolling towards a release. I’ve been talking with Glenn Block quite a bit about Prism, and am even on the advisory board (what were they thinking?).

One of the topics not covered by Prism is occasional connectivity, and I would like to say a word or two about that. First of all, if you’re building a standalone client (one that doesn’t communicate with anything), then there’s a good chance that Prism isn’t for you, although you could be composing other standalone client modules. So, if your client isn’t communicating with anything, well, then this post probably won’t interest you that much. Let’s start with…


Networks fail. Period.

This means that your client machine will not always be connected to other servers.

Also, servers fail – critical Windows patches and just regular power outages.

Ergo, your “smart” client will be occasionally connected, whether you planned for it or not.

And please don’t take this post as a “dumping on Prism” post – it isn’t intended that way. Rather, it is about how you should think about designing modules in Prism, and why.

Modules and Connectivity

Consider the case where we have two modules being composed in a single client. Each module communicates with a different server. Let’s call these modules Ma and Mb, and the servers Sa and Sb respectively. Now, let’s discuss what occurs given that the modules weren’t designed with occasional connectivity in mind.

User clicks something in Mb which requires communication.

Mb tries to call Sb, say, over HTTP, using a regular web service invocation.

The calling thread, in this case, the one used for user interaction, is blocked waiting for a response from Sb.image

Sometime in this call, Sb fails, connectivity goes down, whatever.

30 seconds after the call, the HTTP connection times out.

If something important were happening in Ma at the same time, the user couldn’t even see it, let alone do anything about it since the user interaction thread is stuck. This is a serious concern for the financial services domain, but in many others as well.

You mean there’s more?

I can go on, but I think that that’s enough to paint the picture that if you are building a smart client, there are a lot more things to think about than just learning Prism. That’s my main concern after witnessing what happened around the CAB. Given the learning curve around these frameworks many developers don’t seek to deepen their understanding beyond just becoming proficient with them. This isn’t just centered on the developers, evangelists in Microsoft tend to paint the picture this way:

Once you understand X (CAB, Prism, BizTalk, whatever), all your problems are solved.

That’s not to say there aren’t good things in those technologies, but that’s just it, they’re just tools. Silver hammers and “laser” guided saws do not a master carpenter make. There’s actually a pretty good chance the regular guy will saw their arm off.


I do hope more “instruction manuals” will be coming out of Microsoft on these topics. That’s not to say there aren’t any. Specifically on the topic of occasional connectivity, there is Chapter 4 of the Smart Client Architecture & Design Guide. Unfortunately, it doesn’t say anything about how that connects with the MVC/MVP being used client side (the bits affected by Prism). Chapter 6 of the same guide deals with the client-side threading, but doesn’t address issues like:

  • Which model object instance are views bound to.
  • Do other threads have access to that object at the same time.
  • Which controller/presenter is responsible for giving that object to the view.
  • Do they need to clone it.
  • How deep should the clone be.
  • How do various controllers/presenters (which may be showing the same object in different views at the same time) communicate changes to their various independent clones.

I haven’t yet documented all the patterns that answer these questions, but until I do (or Microsoft does), let me offer these few resources which I’ve put out over the years:

There’s also some more links under the Smart Client link of my “First time here?” page.

Also, please join me in asking Microsoft for an update to these guides – comments below or your own blog posts would be great.

WCF, Smart Clients, and Deadlocks

Friday, April 11th, 2008

There’s a new article up on MSDN describing how to write Smart Clients using WCF. The author is none other than WCF-Master Lowy and he goes over the multitude of ways you can deadlock yourself.

Here’s a taste:

UI Thread and Concurrency Management

Whenever you use hosting on the UI thread, deadlocks are possible. For example, the following setup is guaranteed to result with a deadlock: A Windows Forms application is hosting a service with UseSynchronizationContext set to true, and UI thread affinity is established. The Windows Forms application then calls the service over one of its endpoints. The call to the service blocks the UI thread, while WCF posts a message to the UI thread to invoke the service. That message is never processed, because of the blocking UI thread—hence, the deadlock.

Another possible case for a deadlock occurs when a Windows Forms application is hosting a service with UseSynchronizationContext set to true and UI thread affinity is established. The service receives a call from a remote client. That call is marshaled to the UI thread and is eventually executed on that thread. If the service is allowed to call out to another service, that can result in a deadlock if the callout causality tries somehow to update the UI or call back to the service’s endpoint, because all of the service instances that are associated with any endpoint (regardless of the service-instancing mode) share the same UI thread.

Similarly, you risk a deadlock if the service is configured for reentrancy and it calls back to its client. You risk a deadlock if the callback causality tries to update the UI or enter the service, because that reentrance must be marshaled to the blocked UI thread.

Actually, I have difficulty believing that Juval would go so far as to suggest that even the forms should be services, but he does:

Form as a Service

The main motivation for hosting a WCF service on the UI thread is if the service must update the UI or the form. The problem is always: How does the service reach out and obtain a reference to the form? While the techniques and ideas that appear thus far in the listings certainly work, it would be simpler yet if the form were the service and hosted itself. For this to work, the form (or any window) must be a singleton service. The reason is that singleton is the only instancing mode that enables you to provide WCF with a live instance to host. In addition, you would not want a per-call form that exists only during a client call (which is usually very brief), nor would you want a per-session form that only a single client can establish a session with and update.

When a form is also a service, having that form as a singleton service is the best instancing mode all around.

I think that this article serves as a great treatise leading to only one conclusion – you’d have to be crazy to try to do this without some higher level framework, preferably with a different low-level framework too 🙂 . Sucks Microsoft didn’t put one out – nor is there a pending beta, CTP, or even word about some project with a codename handling this. From what I know about Prism, it doesn’t intend to handle this issue either.

One thing that isn’t covered in the article is that if you do choose not to tie the client-side service to the UI thread, you open yourself up to race conditions. Reasons you’d want to handle messages on a different thread center around UI responsiveness. I’ve written about these things before:

The more I read things like this, the more I feel that I have to get going with my nServiceBus based solution. I’m fairly swamped as it is, so if anyone is interested in helping get this project off the ground, I’d be most grateful – as I think anyone else that had to build a smart client would.

[Podcast] Message Ordering: Is it Cost Effective?

Tuesday, January 1st, 2008

In this podcast we’ll be discussing the issues around multi-threaded processing of messages by a service, specifically that the processing of message received second may be finished before that of the first. This scenario tends to rear its ugly head at higher levels of load and is critical for correctness in high-scalability environments.

Our long time listener Bill asks:

Hi Udi,

I have a question around processing of messages in proper order. When leveraging multiple threads to process messages in a message queue, it is possible for the second message in the queue to get processed before the first – especially if the first message is considerably larger than the second. I have taken a lot of care to make sure that messages are sent in the correct order, only to find that the receiving system can process them out of order anyway.

Consider a Policy Created notification, which must come before a Policy Approved notification. If both messages are sitting in the queue when the receiving service starts up, the approval message can be processed before the creation message. How can I make sure that message ordering is respected by the receiving system? I am using WCF/MSMQ as the underlying transport by the way. The only way I have found so far is to limit the receiving service to a single thread, which is by no means desirable.

Best Regards,



Download directly here.

Additional References

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

What Makes Smart Clients Safe?

Wednesday, December 26th, 2007

doctor_motion2 After my recent post on using AOP for smart client development, my partner-in-crime, Arnon, suggested I explain a little bit more on the whole issue of multi-threading in the UI. This isn’t going to be another tired explanation of how you should only update controls on the main thread. This is going to be a post on the challenges multi-threading brings and how to address them.

Multiple Threads – Why?

One of the properties of smart clients is that they should be able to work offline. Sometimes that means an explicit action of taking data and bringing it to the local machine so that the user can work on it, and other times it just has to do with the fact that wireless connectivity can be flaky. More interesting scenarios include the submission of batch jobs and receiving notification on when they complete. The bottom line is that the user should be able to continue doing their interactive work uninterrupted as all this is occurring.

While the user is disconnected, obviously the data they are working on is local – the client is not calling the server to perform the work on its behalf. However, in the flaky connectivity scenario, this happens all the time. In other words, the design should be the same for handling all scenarios. One thread for interacting with the user, and (at least) on other thread for handling the issues of connectivity. The one thing that is clear is that we are going to have data on the client with which the user interacts on one thread, and which the background thread will also be updating as notifications arrive from the server.

Local Data and Multi-Threading – a Recipe for Disaster

When multiple threads are working with the same data, unless some specific code is in that object to make it thread-safe, there’s a good chance that object will end up in an inconsistent state. This can be catastrophic if we’re talking about air traffic control systems, dispensing medication, factory floor automation, etc. The problem is that we can’t just lock down the entire system whenever something needs to be done. While in some cases I’ve seen projects make each object thread-safe, implementing some base class for handling locking, that doesn’t work between objects and results in deadlocks.

Just as an example of the multi-object problem, consider the doctor prescribing medication for a patient. Now, the data about a patient in a smart client is not all on the Patient Object – there are lists of connected test results, medication already being taken, which nurses and doctors have treated this patient so far, etc. As a test result gets pushed to the client from the server, a new doctor unfamiliar with the history of the patient orders the same test again. These two objects under Patient – TestRequest and TestResult are being acted upon by different threads. One of the goals of the system was to eliminate duplicate testing for patients – cited as costing the hospital chain millions of dollars a year. Good thing we addressed the cross-object multi-threaded locking thing properly 🙂

Always Switch from Background to Foreground?

One solution I’ve seen used successfully is to always change threads from background to foreground whenever an update comes in from the server. This works when you don’t have many updates or the work performed on the client as a result of an update doesn’t take long. Always keep in mind, though, the flaky connectivity scenario. What tends to happen is that server notifications bunch up and then hit the client all together. When this occurs, clients developed this way are rendered unusable.

It would seem like this solution isn’t valid because of the above, but don’t dismiss it so quickly. This is an easy solution to implement that may address your specific environment and, as such, be cost effective. A large part of the work I’ve been doing is to make the more complex environments just as easy to develop as these simpler ones.

Infrastructure-level, Safe, Multi-Object Locking

Luckily for us developers, in the .net framework there is a class that handles this for us – well, 2 actually: ContextBoundObject and SynchronizationAttribute. When using these two classes, we can create something known as a Synchronization Domain which acts as a global lock for all objects belonging to the synchronization domain. What this means is that if the user thread is trying to add a test request object while the background thread is already creating a test result object, the user thread will block automatically until the background thread completes its work.

There is only one teensy-weensy problem – ContextBoundObjects are really heavy-weight. The last thing you want is having millions of these running around in your client – you’ll end up with a multi-threading safe unusable UI. Also, the creation of an object inheriting from ContextBoundObject takes quite a bit longer than a plain-old.net object. In other words, technology by itself will not solve our problems – we need some patterns for the correct use of the technology so that we can maintain a reasonable level of performance while taking care of safety.

MVC and Threading – Controllers

Regardless of which flavour of MVC you prefer (I’m in the Supervising Controller camp for smart clients), the logic controlling what goes on in the client is found in the controllers. What this means is that actions from the user as well as background notifications will need to go through these controllers. It is important that these controllers be thread-safe since they are state-full – managing which windows are open, which step in a given process a user is currently doing, etc.

The characteristics of these controller objects which make them best suited to inherit from ContextBoundObject are that there are only a handful of these objects at any point in time and that they are created at startup – they’re singletons (in the “only-one-of-them” sense of the word).

The only special thing that controllers need to do in terms of threading is to dispatch calls to view objects on the foreground thread, even if the thread currently running is the background thread. For example, popping up a “toast” that a test result has arrived when a notification from the server comes in.

These elements – inheriting from ContextBoundObject, use of the SynchronizationAttribute, and thread-switching can be pulled up in to a BaseController class:

using System;
using System.ComponentModel;
using System.Runtime.Remoting.Contexts;

namespace ControllerFramework
    public class BaseController : ContextBoundObject
        protected ISynchronizeInvoke invoker;
        public ISynchronizeInvoke Invoker
            get { return invoker; }
            set { invoker = value; }

        protected void MarshalToUiThread(Delegate toCall, params object[] parameters)
            if (this.invoker == null)

            if (this.invoker.InvokeRequired)
                invoker.BeginInvoke(toCall, parameters);

Well, I think that this is long enough for a single blog post. In the next instalment of this series I’ll be talking about how model objects and views fit into the multi-threaded smart client. After that, we’ll be seeing how service agents, messaging, and service contract design need to be done in this style. While all this blogging will be going on, I’ll be getting a software factory up that will tie all these patterns and frameworks together so that all developers will be able to write thread-safe, high-performance smart clients without needing a doctorate in computer science – not that I have one 🙂

Questions? Comments? Thoughts?

Handling messages out of order

Saturday, December 15th, 2007

I wanted to follow up on my recent post, “In order messaging a myth?” by showing the exact code that solves the issue. I have a podcast waiting to come online that deals with the specifics, so keep your eye out for that too.

The important thing to note is that if we just automatically return the message to the queue, we may get “stuck” with that message if the first PolicyCreatedMessage never arrived. This opens us up to a Denial-of-Service attack by quite simply flooding us with a bunch of messages that never get cleaned up.

Anyway, the general idea is to first try the regular happy path, and only if we see that prerequisite data isn’t available, do we see if another thread may be working on that data. This is done by decreasing the isolation level of our transaction from the regular ReadCommitted to ReadUncommitted. This will enable our thread to see if some other thread inserted the policy in to the Policies table but hasn’t committed its transaction yet.

    public class PolicyApprovedMessageHandler : BaseDBMessageHandler<PolicyApprovedMessage>


        public override void Handle(PolicyApprovedMessage message)


            bool policyExists = true;


            using (ISession s = OpenSession())

            using (ITransaction tx = s.BeginTransaction(IsolationLevel.ReadCommitted))


                Policy p = s.Get<Policy>(message.PolicyId);


                if (p != null)






                    policyExists = false;



            if (!policyExists) // check to make sure

                using (ISession s = OpenSession())

                using (ITransaction tx = s.BeginTransaction(IsolationLevel.ReadUncommitted))


                    Policy p = s.Get<Policy>(message.PolicyId);


                    if (p != null) // another thread hasn’t committed its tx yet, so try message again later







The next step will be how we take this code and make it generic, so that we don’t have write the same code over and over again for the different kinds of message handlers we have.

But that will have to wait until the next installment 🙂

Eureka! AOP is the final piece of the multi-threaded smart client puzzle

Friday, December 7th, 2007

If you’ve read my recent post on the threading issues I’ve been dealing with in Smart Client Applications, then you’re probably beginning to get the picture that its fairly complex. To tell you the truth, it is. And up until this point I haven’t been able to find anything that’ll help – and that includes the CAB/SCSF. But yesterday I had my epiphany. The answer was in AOP.

You see, the main problem that I hadn’t been able to solve was that in order for the code to be thread-safe, you had to make sure that no code in the views would/could change entity data. One solution is not to use data-binding, which sucks, but isn’t enough to be sure. Another solution is to have all supervising-controllers clone an entity before they give it to a view. Even if you could possibly code review every line of those classes, the new guy (or old guy who forgot) will, by accident, write one new line of code that could pass an entity to a view without cloning it first. That’s not a very sustainable solution.

This thing has been bothering me for a couple of months now and I hadn’t found a way around it. Until yesterday, like I said. I was talking to somebody about threading stuff, and somehow my unconscience lobbed me this thought about AOP. Now I’m not the sharpest pencil in the pack, but I know to listen when my unconscience “speaks”.

So I set about going over what I knew about AOP – interceptors, advisors, advice, introductions, etc, etc. And then it dawned on me. I could intercept all calls to any object that implemented IView, check the parameters of those calls, and if they implemented IEntity, to clone them before passing them through.

<Homer-style WOOHOO />

The great thing is that developers don’t need to remember to clone entities – it happens automatically. The even greater thing is that this will lead developers to writing the correct kind of interaction between their views and supervising controllers.

Together with nServiceBus, this is going to make the extremely difficult problem of writing thread-safe smart clients possible.

I’ve never made use of AOP in a framework before so I’d like to get the broader community’s feedback on this before incorporating this in production. I’ve spoken with some serious AOP folks who have allayed most of my uncertainties, but I’d like to hear more. Anyway, here’s the proof of concept (that makes use of Spring).

If this turns out to be a viable solution, I think we’ll have a solid environment for building a software factory on top of. That is something that I’m really excited about. In this multi-core future (present) that is upon us, multi-threading on the client is pretty much a necessity. We need a way to get things safe and stable by default without requiring a member of the CLR team to hold our hand.

Anybody who’s interested in helping, drop a comment below.

Object Builder – the place to fix system-wide threading bugs

Thursday, December 6th, 2007

Last week I was at a client in their test lab and saw a strange bit of behavior. The system could be described as something like an air traffic control system, showing things moving around on a map. For just a second, a fraction of a second, one of the “planes” disappeared from the map and then reappeared again.

When I asked if anybody else saw it, one of the developers said, “Yeah, that happens sometimes – but it fixes itself right after that.”

“What if the user sends a command to the server making use of that location?”, the PM asked. “Could that cause them to collide?”

You could hear a pin drop.

After everyone got passed the preliminary shock, we got down to work. I asked if I could look at the logs, but after more than an hour, I found nothing. No reason to explain the strange behavior. I suggested doing some more instrumentation so that whenever a location changed on the client-side entities, we’d write that to the log.

After that, we ran the system again in the lab under the expected load (several hundred things moving every second, and the user doing the expected activity) and didn’t notice anything. An intern “volunteered” to keep working the system while the rest of us went to lunch. When we came back, he told us that everything seemed to be working OK.

These Heisenbugs are the things that keep me up at night.

“Watching the system changes its behavior”, one of the older devs nodded his head sagely.

Just as we were about to leave the lab another one of the developers gave a shout, “It did it again!”. We quickly stopped the system. Opened the (rather huge) log files and looked for the latest entries.

There it was.

A context switch between setting the latitude and longitude of an entity.

That should not have happened. Not that context switches don’t happen, but rather that it should have been impossible by design. We had made use of synchronization domains and the appropriate patterns so that two threads could never concurrently be working on the same instance of an entity. The synchronization features baked in to nServiceBus had taken care of everything up to that point.

Before getting into the threading solution, I want to address a specific alternate patch that was deployed in the meantime:

The solution for the long/lat problem was simple – just make Location a value object and use a single setter for it rather than one for Latitide and another for Longitude. We were still worried about other bits of data that were correllated in the domain – things that couldn’t be solved the same way.

After getting 3 grizzled C++ veterans in the room, we did a code walkthrough of the threading model of nServiceBus. We went through the nitty gritty details of synchronization domains, how the Bus object was outside of the domain, why that was important for user experience, how the message handlers couldn’t be ContextBoundObjects because of the performance impact of creating and destroying them at a high rate, why they couldn’t just be singletons, why they still had to run in the synchronization domain, so that the UI thread couldn’t work on the same (or related) objects at the same time, etc, etc.

And then it hit me.

The bus was communicated directly with the message handlers.

After the Object Builder created the message handler, the bus dispatched the message to the handler directly. And since the bus was outside the synchronization domain, then the thread calling into the handler wouldn’t have locked the domain, leaving the UI thread open to go in and touch those very same objects.

They say that really understanding the problem is 90% of the solution. I’m hoping to meet them some day, because they’re really smart.

All that we needed to do was have the Object Builder dispatch the message to the handler instead of the bus – since the builder was configured to be in the synchronization domain (on the client side). Something as simple as just adding the method:

void BuildAndDispatch(Type typeToBuild, string methodName, params object[] methodArgs);

So, instead of the bus using this code:

object handler = this builder.Build(messageHandlerType);
MethodInfo method = messageHandlerType.GetMethod(“Handle”);
method.Invoke(handler, messageToBeDispatched);

It would do:

this.builder.BuildAndDispatch(messageHandlerType, “Handle”, messageToBeDispatched);

[Just FYI, this is now up on the sourceforge site]

We redeployed the system to the lab, ran all the functional, stress, load, etc tests and everything appeared to be stable. The system has been under scrutiny for the past 4 days by batteries of testers instructed specifically to look for those strage kinds of behavior. Other developers are running scripts on the log files looking for other kinds of context switches that may have been missed by the testers. I am happy to report that they haven’t found anything.

Not that this means that the problem isn’t there. We really can’t be sure. However, the PM has decided that we are stable enough to go into pilot mode – deploying into production beside the current system; having users work on both systems at the same time. I’m optimistic.

I’m personally involved in two more production-projects that are making use of nServiceBus in similarly high-end situations and we’ve never had these threading problems – now two years running.

That was an interesting week.

[Podcast] Thread-Safe Asynchronous Smart Clients

Friday, October 12th, 2007

In this podcast we’ll look at various patterns involved in creating MVC-based Smart Clients which communicate using asynchronous messaging and how to avoid threading problems there.

Neil asks:

Hi Udi,

We’re building a smart client application that uses WCF for full-duplex communications with our server. This is the asynchronous communication you talk about in your podcast. The smart-client is based on the MVC pattern, where model objects raise events when they’re changed so that the views can update themselves.

What’s started happening recently is that the smart-client has been freezing-up on us intermittently. We don’t know how to debug this and are wondering if its an architectural problem.

Any help you can give would be most appreciated.



Download via the Dr. Dobb’s site

Or download directly here

Additional References

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

ObjectBuilder synchronization features needed for pub/sub-ing Smart Clients

Friday, September 28th, 2007

I’ve been getting some questions from the Dependency Injection folks out there as to why I have my own Object Builder wrapping the framework. There are two very good reasons why I do this:

The first is to insulate the framework and application code that I write from the choice of one dependency injection technology or another. I want the ability to switch easily from one to the other – not so much that projects go back and forth. Updating those config files is definitely not easy. However, it allows me to have “portable” framework code that is applicable to all the projects I consult on, regardless of their choice of technology.

The second has to do with NServiceBus specifically. In order to make use of duplex communication on smart clients, you need a background thread. That thread will be updating the same (model) objects as the UI thread. That means we need synchronization. I prefer to use .NET’s built-in synchronization domains in order to solve this rather thorny problem.

The only thing is that message handlers need to be in the synchronization domain so that they can easily update those objects. However, the Bus object must not be in the synchronization domain so that if we’ve received a large update from the server, we won’t be locking out the UI thread from interacting with data on the client.

Since the bus makes use of a dependency injection framework to create message handlers, this was the best place to put the code which causes message handlers to run within the synchronization domain.

Be aware that in order to enjoy this feature, you need to split up those large server updates into multiple, logical objects (that implement IMessage), but you can still publish them all in one go using the method:

void Publish(params IMessage[] messages);

And, of course, you need to set the JoinSynchronizationDomain property of the Object Builder.

I’ll have a podcast coming out on this topic soon.

You can get the code here:

Object Builder.zip

But you’ll have to get the Spring Framework code from the official site. Make sure you download RC 1.1. Then, take the binaries and copy them to the “BIN” folder of the Object Builder solution. If you’re looking to save on some “weight”, you only need “Spring.Core.dll”, “Common.Logging.dll” and “antlr.runtime.dll” for the solution to compile. You will need one of the logging implementations DLLs to get anything written to a log, obviously.


Don't miss my best content


Bryan Wheeler, Director Platform Development at msnbc.com
Udi Dahan is the real deal.

We brought him on site to give our development staff the 5-day “Advanced Distributed System Design” training. The course profoundly changed our understanding and approach to SOA and distributed systems.

Consider some of the evidence: 1. Months later, developers still make allusions to concepts learned in the course nearly every day 2. One of our developers went home and made her husband (a developer at another company) sign up for the course at a subsequent date/venue 3. Based on what we learned, we’ve made constant improvements to our architecture that have helped us to adapt to our ever changing business domain at scale and speed If you have the opportunity to receive the training, you will make a substantial paradigm shift.

If I were to do the whole thing over again, I’d start the week by playing the clip from the Matrix where Morpheus offers Neo the choice between the red and blue pills. Once you make the intellectual leap, you’ll never look at distributed systems the same way.

Beyond the training, we were able to spend some time with Udi discussing issues unique to our business domain. Because Udi is a rare combination of a big picture thinker and a low level doer, he can quickly hone in on various issues and quickly make good (if not startling) recommendations to help solve tough technical issues.” November 11, 2010

Sam Gentile Sam Gentile, Independent WCF & SOA Expert
“Udi, one of the great minds in this area.
A man I respect immensely.”

Ian Robinson Ian Robinson, Principal Consultant at ThoughtWorks
"Your blog and articles have been enormously useful in shaping, testing and refining my own approach to delivering on SOA initiatives over the last few years. Over and against a certain 3-layer-application-architecture-blown-out-to- distributed-proportions school of SOA, your writing, steers a far more valuable course."

Shy Cohen Shy Cohen, Senior Program Manager at Microsoft
“Udi is a world renowned software architect and speaker. I met Udi at a conference that we were both speaking at, and immediately recognized his keen insight and razor-sharp intellect. Our shared passion for SOA and the advancement of its practice launched a discussion that lasted into the small hours of the night.
It was evident through that discussion that Udi is one of the most knowledgeable people in the SOA space. It was also clear why – Udi does not settle for mediocrity, and seeks to fully understand (or define) the logic and principles behind things.
Humble yet uncompromising, Udi is a pleasure to interact with.”

Glenn Block Glenn Block, Senior Program Manager - WCF at Microsoft
“I have known Udi for many years having attended his workshops and having several personal interactions including working with him when we were building our Composite Application Guidance in patterns & practices. What impresses me about Udi is his deep insight into how to address business problems through sound architecture. Backed by many years of building mission critical real world distributed systems it is no wonder that Udi is the best at what he does. When customers have deep issues with their system design, I point them Udi's way.”

Karl Wannenmacher Karl Wannenmacher, Senior Lead Expert at Frequentis AG
“I have been following Udi’s blog and podcasts since 2007. I’m convinced that he is one of the most knowledgeable and experienced people in the field of SOA, EDA and large scale systems.
Udi helped Frequentis to design a major subsystem of a large mission critical system with a nationwide deployment based on NServiceBus. It was impressive to see how he took the initial architecture and turned it upside down leading to a very flexible and scalable yet simple system without knowing the details of the business domain. I highly recommend consulting with Udi when it comes to large scale mission critical systems in any domain.”

Simon Segal Simon Segal, Independent Consultant
“Udi is one of the outstanding software development minds in the world today, his vast insights into Service Oriented Architectures and Smart Clients in particular are indeed a rare commodity. Udi is also an exceptional teacher and can help lead teams to fall into the pit of success. I would recommend Udi to anyone considering some Architecural guidance and support in their next project.”

Ohad Israeli Ohad Israeli, Chief Architect at Hewlett-Packard, Indigo Division
“When you need a man to do the job Udi is your man! No matter if you are facing near deadline deadlock or at the early stages of your development, if you have a problem Udi is the one who will probably be able to solve it, with his large experience at the industry and his widely horizons of thinking , he is always full of just in place great architectural ideas.
I am honored to have Udi as a colleague and a friend (plus having his cell phone on my speed dial).”

Ward Bell Ward Bell, VP Product Development at IdeaBlade
“Everyone will tell you how smart and knowledgable Udi is ... and they are oh-so-right. Let me add that Udi is a smart LISTENER. He's always calibrating what he has to offer with your needs and your experience ... looking for the fit. He has strongly held views ... and the ability to temper them with the nuances of the situation.
I trust Udi to tell me what I need to hear, even if I don't want to hear it, ... in a way that I can hear it. That's a rare skill to go along with his command and intelligence.”

Eli Brin, Program Manager at RISCO Group
“We hired Udi as a SOA specialist for a large scale project. The development is outsourced to India. SOA is a buzzword used almost for anything today. We wanted to understand what SOA really is, and what is the meaning and practice to develop a SOA based system.
We identified Udi as the one that can put some sense and order in our minds. We started with a private customized SOA training for the entire team in Israel. After that I had several focused sessions regarding our architecture and design.
I will summarize it simply (as he is the software simplist): We are very happy to have Udi in our project. It has a great benefit. We feel good and assured with the knowledge and practice he brings. He doesn’t talk over our heads. We assimilated nServicebus as the ESB of the project. I highly recommend you to bring Udi into your project.”

Catherine Hole Catherine Hole, Senior Project Manager at the Norwegian Health Network
“My colleagues and I have spent five interesting days with Udi - diving into the many aspects of SOA. Udi has shown impressive abilities of understanding organizational challenges, and has brought the business perspective into our way of looking at services. He has an excellent understanding of the many layers from business at the top to the technical infrstructure at the bottom. He is a great listener, and manages to simplify challenges in a way that is understandable both for developers and CEOs, and all the specialists in between.”

Yoel Arnon Yoel Arnon, MSMQ Expert
“Udi has a unique, in depth understanding of service oriented architecture and how it should be used in the real world, combined with excellent presentation skills. I think Udi should be a premier choice for a consultant or architect of distributed systems.”

Vadim Mesonzhnik, Development Project Lead at Polycom
“When we were faced with a task of creating a high performance server for a video-tele conferencing domain we decided to opt for a stateless cluster with SQL server approach. In order to confirm our decision we invited Udi.

After carefully listening for 2 hours he said: "With your kind of high availability and performance requirements you don’t want to go with stateless architecture."

One simple sentence saved us from implementing a wrong product and finding that out after years of development. No matter whether our former decisions were confirmed or altered, it gave us great confidence to move forward relying on the experience, industry best-practices and time-proven techniques that Udi shared with us.
It was a distinct pleasure and a unique opportunity to learn from someone who is among the best at what he does.”

Jack Van Hoof Jack Van Hoof, Enterprise Integration Architect at Dutch Railways
“Udi is a respected visionary on SOA and EDA, whose opinion I most of the time (if not always) highly agree with. The nice thing about Udi is that he is able to explain architectural concepts in terms of practical code-level examples.”

Neil Robbins Neil Robbins, Applications Architect at Brit Insurance
“Having followed Udi's blog and other writings for a number of years I attended Udi's two day course on 'Loosely Coupled Messaging with NServiceBus' at SkillsMatter, London.

I would strongly recommend this course to anyone with an interest in how to develop IT systems which provide immediate and future fitness for purpose. An influential and innovative thought leader and practitioner in his field, Udi demonstrates and shares a phenomenally in depth knowledge that proves his position as one of the premier experts in his field globally.

The course has enhanced my knowledge and skills in ways that I am able to immediately apply to provide benefits to my employer. Additionally though I will be able to build upon what I learned in my 2 days with Udi and have no doubt that it will only enhance my future career.

I cannot recommend Udi, and his courses, highly enough.”

Nick Malik Nick Malik, Enterprise Architect at Microsoft Corporation
You are an excellent speaker and trainer, Udi, and I've had the fortunate experience of having attended one of your presentations. I believe that you are a knowledgable and intelligent man.”

Sean Farmar Sean Farmar, Chief Technical Architect at Candidate Manager Ltd
“Udi has provided us with guidance in system architecture and supports our implementation of NServiceBus in our core business application.

He accompanied us in all stages of our development cycle and helped us put vision into real life distributed scalable software. He brought fresh thinking, great in depth of understanding software, and ongoing support that proved as valuable and cost effective.

Udi has the unique ability to analyze the business problem and come up with a simple and elegant solution for the code and the business alike.
With Udi's attention to details, and knowledge we avoided pit falls that would cost us dearly.”

Børge Hansen Børge Hansen, Architect Advisor at Microsoft
“Udi delivered a 5 hour long workshop on SOA for aspiring architects in Norway. While keeping everyone awake and excited Udi gave us some great insights and really delivered on making complex software challenges simple. Truly the software simplist.”

Motty Cohen, SW Manager at KorenTec Technologies
“I know Udi very well from our mutual work at KorenTec. During the analysis and design of a complex, distributed C4I system - where the basic concepts of NServiceBus start to emerge - I gained a lot of "Udi's hours" so I can surely say that he is a professional, skilled architect with fresh ideas and unique perspective for solving complex architecture challenges. His ideas, concepts and parts of the artifacts are the basis of several state-of-the-art C4I systems that I was involved in their architecture design.”

Aaron Jensen Aaron Jensen, VP of Engineering at Eleutian Technology
Awesome. Just awesome.

We’d been meaning to delve into messaging at Eleutian after multiple discussions with and blog posts from Greg Young and Udi Dahan in the past. We weren’t entirely sure where to start, how to start, what tools to use, how to use them, etc. Being able to sit in a room with Udi for an entire week while he described exactly how, why and what he does to tackle a massive enterprise system was invaluable to say the least.

We now have a much better direction and, more importantly, have the confidence we need to start introducing these powerful concepts into production at Eleutian.”

Gad Rosenthal Gad Rosenthal, Department Manager at Retalix
“A thinking person. Brought fresh and valuable ideas that helped us in architecting our product. When recommending a solution he supports it with evidence and detail so you can successfully act based on it. Udi's support "comes on all levels" - As the solution architect through to the detailed class design. Trustworthy!”

Chris Bilson Chris Bilson, Developer at Russell Investment Group
“I had the pleasure of attending a workshop Udi led at the Seattle ALT.NET conference in February 2009. I have been reading Udi's articles and listening to his podcasts for a long time and have always looked to him as a source of advice on software architecture.
When I actually met him and talked to him I was even more impressed. Not only is Udi an extremely likable person, he's got that rare gift of being able to explain complex concepts and ideas in a way that is easy to understand.
All the attendees of the workshop greatly appreciate the time he spent with us and the amazing insights into service oriented architecture he shared with us.”

Alexey Shestialtynov Alexey Shestialtynov, Senior .Net Developer at Candidate Manager
“I met Udi at Candidate Manager where he was brought in part-time as a consultant to help the company make its flagship product more scalable. For me, even after 30 years in software development, working with Udi was a great learning experience. I simply love his fresh ideas and architecture insights.
As we all know it is not enough to be armed with best tools and technologies to be successful in software - there is still human factor involved. When, as it happens, the project got in trouble, management asked Udi to step into a leadership role and bring it back on track. This he did in the span of a month. I can only wish that things had been done this way from the very beginning.
I look forward to working with Udi again in the future.”

Christopher Bennage Christopher Bennage, President at Blue Spire Consulting, Inc.
“My company was hired to be the primary development team for a large scale and highly distributed application. Since these are not necessarily everyday requirements, we wanted to bring in some additional expertise. We chose Udi because of his blogging, podcasting, and speaking. We asked him to to review our architectural strategy as well as the overall viability of project.
I was very impressed, as Udi demonstrated a broad understanding of the sorts of problems we would face. His advice was honest and unbiased and very pragmatic. Whenever I questioned him on particular points, he was able to backup his opinion with real life examples. I was also impressed with his clarity and precision. He was very careful to untangle the meaning of words that might be overloaded or otherwise confusing. While Udi's hourly rate may not be the cheapest, the ROI is undoubtedly a deal. I would highly recommend consulting with Udi.”

Robert Lewkovich, Product / Development Manager at Eggs Overnight
“Udi's advice and consulting were a huge time saver for the project I'm responsible for. The $ spent were well worth it and provided me with a more complete understanding of nServiceBus and most importantly in helping make the correct architectural decisions earlier thereby reducing later, and more expensive, rework.”

Ray Houston Ray Houston, Director of Development at TOPAZ Technologies
“Udi's SOA class made me smart - it was awesome.

The class was very well put together. The materials were clear and concise and Udi did a fantastic job presenting it. It was a good mixture of lecture, coding, and question and answer. I fully expected that I would be taking notes like crazy, but it was so well laid out that the only thing I wrote down the entire course was what I wanted for lunch. Udi provided us with all the lecture materials and everyone has access to all of the samples which are in the nServiceBus trunk.

Now I know why Udi is the "Software Simplist." I was amazed to find that all the code and solutions were indeed very simple. The patterns that Udi presented keep things simple by isolating complexity so that it doesn't creep into your day to day code. The domain code looks the same if it's running in a single process or if it's running in 100 processes.”

Ian Cooper Ian Cooper, Team Lead at Beazley
“Udi is one of the leaders in the .Net development community, one of the truly smart guys who do not just get best architectural practice well enough to educate others but drives innovation. Udi consistently challenges my thinking in ways that make me better at what I do.”

Liron Levy, Team Leader at Rafael
“I've met Udi when I worked as a team leader in Rafael. One of the most senior managers there knew Udi because he was doing superb architecture job in another Rafael project and he recommended bringing him on board to help the project I was leading.
Udi brought with him fresh solutions and invaluable deep architecture insights. He is an authority on SOA (service oriented architecture) and this was a tremendous help in our project.
On the personal level - Udi is a great communicator and can persuade even the most difficult audiences (I was part of such an audience myself..) by bringing sound explanations that draw on his extensive knowledge in the software business. Working with Udi was a great learning experience for me, and I'll be happy to work with him again in the future.”

Adam Dymitruk Adam Dymitruk, Director of IT at Apara Systems
“I met Udi for the first time at DevTeach in Montreal back in early 2007. While Udi is usually involved in SOA subjects, his knowledge spans all of a software development company's concerns. I would not hesitate to recommend Udi for any company that needs excellent leadership, mentoring, problem solving, application of patterns, implementation of methodologies and straight out solution development.
There are very few people in the world that are as dedicated to their craft as Udi is to his. At ALT.NET Seattle, Udi explained many core ideas about SOA. The team that I brought with me found his workshop and other talks the highlight of the event and provided the most value to us and our organization. I am thrilled to have the opportunity to recommend him.”

Eytan Michaeli Eytan Michaeli, CTO Korentec
“Udi was responsible for a major project in the company, and as a chief architect designed a complex multi server C4I system with many innovations and excellent performance.”

Carl Kenne Carl Kenne, .Net Consultant at Dotway AB
“Udi's session "DDD in Enterprise apps" was truly an eye opener. Udi has a great ability to explain complex enterprise designs in a very comprehensive and inspiring way. I've seen several sessions on both DDD and SOA in the past, but Udi puts it in a completly new perspective and makes us understand what it's all really about. If you ever have a chance to see any of Udi's sessions in the future, take it!”

Avi Nehama, R&D Project Manager at Retalix
“Not only that Udi is a briliant software architecture consultant, he also has remarkable abilities to present complex ideas in a simple and concise manner, and...
always with a smile. Udi is indeed a top-league professional!”

Ben Scheirman Ben Scheirman, Lead Developer at CenterPoint Energy
“Udi is one of those rare people who not only deeply understands SOA and domain driven design, but also eloquently conveys that in an easy to grasp way. He is patient, polite, and easy to talk to. I'm extremely glad I came to his workshop on SOA.”

Scott C. Reynolds Scott C. Reynolds, Director of Software Engineering at CBLPath
“Udi is consistently advancing the state of thought in software architecture, service orientation, and domain modeling.
His mastery of the technologies and techniques is second to none, but he pairs that with a singular ability to listen and communicate effectively with all parties, technical and non, to help people arrive at context-appropriate solutions. Every time I have worked with Udi, or attended a talk of his, or just had a conversation with him I have come away from it enriched with new understanding about the ideas discussed.”

Evgeny-Hen Osipow, Head of R&D at PCLine
“Udi has helped PCLine on projects by implementing architectural blueprints demonstrating the value of simple design and code.”

Rhys Campbell Rhys Campbell, Owner at Artemis West
“For many years I have been following the works of Udi. His explanation of often complex design and architectural concepts are so cleanly broken down that even the most junior of architects can begin to understand these concepts. These concepts however tend to typify the "real world" problems we face daily so even the most experienced software expert will find himself in an "Aha!" moment when following Udi teachings.
It was a pleasure to finally meet Udi in Seattle Alt.Net OpenSpaces 2008, where I was pleasantly surprised at how down-to-earth and approachable he was. His depth and breadth of software knowledge also became apparent when discussion with his peers quickly dove deep in to the problems we current face. If given the opportunity to work with or recommend Udi I would quickly take that chance. When I think .Net Architecture, I think Udi.”

Sverre Hundeide Sverre Hundeide, Senior Consultant at Objectware
“Udi had been hired to present the third LEAP master class in Oslo. He is an well known international expert on enterprise software architecture and design, and is the author of the open source messaging framework nServiceBus. The entire class was based on discussion and interaction with the audience, and the only Power Point slide used was the one showing the agenda.
He started out with sketching a naive traditional n-tier application (big ball of mud), and based on suggestions from the audience we explored different solutions which might improve the solution. Whatever suggestions we threw at him, he always had a thoroughly considered answer describing pros and cons with the suggested solution. He obviously has a lot of experience with real world enterprise SOA applications.”

Raphaël Wouters Raphaël Wouters, Owner/Managing Partner at Medinternals
“I attended Udi's excellent course 'Advanced Distributed System Design with SOA and DDD' at Skillsmatter. Few people can truly claim such a high skill and expertise level, present it using a pragmatic, concrete no-nonsense approach and still stay reachable.”

Nimrod Peleg Nimrod Peleg, Lab Engineer at Technion IIT
“One of the best programmers and software engineer I've ever met, creative, knows how to design and implemet, very collaborative and finally - the applications he designed implemeted work for many years without any problems!

Jose Manuel Beas
“When I attended Udi's SOA Workshop, then it suddenly changed my view of what Service Oriented Architectures were all about. Udi explained complex concepts very clearly and created a very productive discussion environment where all the attendees could learn a lot. I strongly recommend hiring Udi.”

Daniel Jin Daniel Jin, Senior Lead Developer at PJM Interconnection
“Udi is one of the top SOA guru in the .NET space. He is always eager to help others by sharing his knowledge and experiences. His blog articles often offer deep insights and is a invaluable resource. I highly recommend him.”

Pasi Taive Pasi Taive, Chief Architect at Tieto
“I attended both of Udi's "UI Composition Key to SOA Success" and "DDD in Enterprise Apps" sessions and they were exceptionally good. I will definitely participate in his sessions again. Udi is a great presenter and has the ability to explain complex issues in a manner that everyone understands.”

Eran Sagi, Software Architect at HP
“So far, I heard about Service Oriented architecture all over. Everyone mentions it – the big buzz word. But, when I actually asked someone for what does it really mean, no one managed to give me a complete satisfied answer. Finally in his excellent course “Advanced Distributed Systems”, I got the answers I was looking for. Udi went over the different motivations (principles) of Services Oriented, explained them well one by one, and showed how each one could be technically addressed using NService bus. In his course, Udi also explain the way of thinking when coming to design a Service Oriented system. What are the questions you need to ask yourself in order to shape your system, place the logic in the right places for best Service Oriented system.

I would recommend this course for any architect or developer who deals with distributed system, but not only. In my work we do not have a real distributed system, but one PC which host both the UI application and the different services inside, all communicating via WCF. I found that many of the architecture principles and motivations of SOA apply for our system as well. Enough that you have SW partitioned into components and most of the principles becomes relevant to you as well. Bottom line – an excellent course recommended to any SW Architect, or any developer dealing with distributed system.”

Consult with Udi

Guest Authored Books

Creative Commons License  © Copyright 2005-2011, Udi Dahan. email@UdiDahan.com