A couple of months ago I put out a post discussing one way to implement custom fetching strategies. Anyway, I finally got around to putting my money where my mouth was…

So, I’ve implemented the pattern in NHibernate, adding the following method to ISession:

As well as adding the following interface to the NHibernate package:

All this enables you to have a stronger separation between your service layer classes and your domain model class, as well as for you to express each service-level use case as a domain concept – an interface.

Once you have such an interface, you can create a fetching strategy for that use case and define exactly how deep of an object graph you want to load so that you only hit the DB once for that use case.

The nice thing is that its all configured with Spring. In other words, if you the entry for your fetching strategy class exists, you get the improved performance, if it doesn’t, you don’t. All without touching your service layer classes.

Just as an example, when I’m in the use case modeled by “ICustomer”, I want to get all the customer’s orders, and their orderlines. This would be done by having a class like this:

And the configuration would look like this (as a part of the regular spring template):

If you want to take a look at the full solution, you can find it here. For some reason, the combined file was too big for the upload on my blog so it’s split into two. Unzip both packages into the same directory. You’ll find a file called “db_scripts.sql” which contains the schema for the DB. Don’t forget to update your connection string in the “hibernate.cfg.xml”. If you’re looking for the changes I made to the NHibernate source, you can find it in the “Updated NHibernate Files” directory. The only real change is to the “SessionImpl.cs” file.

Relevant NHibernate and Spring binaries.

Source code of example.

BTW, there is some intelligent thread-safe caching going on in SessionImpl now so that you get a much smaller performance hit (in terms of code that uses reflection) on subsequent usages of the same interfaces.

Let me know what you think.

In this podcast we’ll try to describe some of the pitfalls of trying to split a domain model between multiple services, as well as how SOA side-steps the “single version of the truth” issue found in reporting.

Rishi asks:

Hi Udi,
First of all, thanks for all the posts and info you share, it is very insightful compared to loads n’ loads of marketing bull and vendor whitepapers. Ok, I have two questions for you.

1. I want to create a SOA-based LOB application/platform and I generally understand the ‘tenets of services’ and reasons for the same. However, as you may well know, there is a lot of interdependency between business constructs (such as a customer with an order, or a delivery with a product construct). Now, should we share these constructs within a tightly-coupled and domain-based “system” which exposes a number of services to the outside world and use the constructs directly inside the so-called “system” boundary. Or on the opposite spectrum have a number of autonomous and independent services that expose, own, and share certain business constructs or at least part of the data that represents them as a whole. For example, an order service will need to interact with a customer construct, which primarily/essentially is (or should be?) owned by a customer profile service – in this context, with clear, direct, and immediate inter-dependency what is preferable? Where do we draw the enclosing line?

2. In a line of business application, reporting and shaping data are a given necessity. Now, with autonomous and independent services which define business constructs in their contextual ways, how do we practically shape, combine, and filter data across various services? We certainly can’t do joins over multiple services in a practical way? And as some suggest, if we are to maintain duplicate or master data elsewhere, it just seems very messy to me – just consider having 3 versions of an order services, with 2 versions of support services, and ‘n’ number of other services to comb data from. Further, for reporting and also analytical purposes we really need to have business data structured in a well-defined manner, which for all purposes needs to be the “single version of the truth”. I can’t see granular services, with all its tenets, sitting nicely with such other requirements of the business – and yet I am not even asking them to be real-time. How do we meet such business requirements in your view with SOA?

Hope the above makes sense!

Cheers,
Rishi

PS: I use the word business construct purposefully, and it does not directly attribute to a well-defined programmable entity (in object-oriented way, that is).

Download via the Dr. Dobb’s site.

Or download directly here.

Additional References

• Ask Udi Podcast #000–Does SOA mean the end of OO?
• Ask Udi Podcast #009–Master Data Management and SOA
• Ask Udi Podcast #016–How does Extract, Transform, Load fit with SOA?

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

This week we’re discussing scenarios involving the use of multiple autonomous components handling the same event. We also get into the topics of component hosting as well as solution development structure.

Our long-time listener Bill asks the following:

Hi Udi,

Thank you for your podcast clarifying the concept of autonomous components. It certainly cleared up a lot in terms of how you define an autonomous component and helped position the concept better in my mind.

I am considering building our Policy Administration service with 3 autonomous components as you previously suggested – one for each insurance product family. As you previously mentioned, these autonomous components could either share a database instance or have separate database instances, even though they may share schema (being a part of the same service).

What concerns me however is that there is some data that must be shared between these autonomous components. A perfect example is Region. If a new Region is defined somewhere in the enterprise, a RegionCreatedNotification is published onto the bus. If I have 3 autonomous components sharing the same database, is it best I just choose one to be the subscriber for this event? Or should I create a new separate queue to receive these notifications and put a message handler for it in a separate MessageHandlers assembly? Or should I subscribe all 3 autonomous components and have them each check if the Region has already been created in the database before inserting a new one?

Option (1) seems presumptuous because it assumes all 3 autonomous components are sharing the same database. Option (2) seems a bit strange because it is like creating a 4th autonomous component only to receive the RegionCreatedNotification messages. And Option (3) seems wasteful since there is redundant code and the system is doing extra unnecessary lookups. At the moment it looks like Option (2) is my best bet, but I’m very interested to get your take on it.

Another question is whether you would say that autonomous components should be hosted in different processes. Since I don’t need to run these autonomous components on separate servers at this stage, it would seem wasteful to create a service host for each component. I could always create separate service hosts at a later time if I needed to host one or more autonomous components on other servers.

And the final question I have is in terms of solution structure. I was contemplating the following structure:

ProductFamily1
———————-
Company.PolicyAdministration.ProductFamily1.Domain
Company.PolicyAdministration.ProductFamily1.MessageHandlers
Company.PolicyAdministration.ProductFamily1.Messages
Company.PolicyAdministration.ProductFamily1.Persistence
Company.PolicyAdministration.ProductFamily1.Persistence.Implementation
Company.PolicyAdministration.ProductFamily1.Etc…

ProductFamily2
———————-
Company.PolicyAdministration.ProductFamily2.Domain
Company.PolicyAdministration.ProductFamily2.MessageHandlers
Company.PolicyAdministration.ProductFamily2.Messages
Company.PolicyAdministration.ProductFamily2.Persistence
Company.PolicyAdministration.ProductFamily2.Persistence.Implementation
Company.PolicyAdministration.ProductFamily2.Etc…

ProductFamily3
———————-
Company.PolicyAdministration.ProductFamily3.Domain
Company.PolicyAdministration.ProductFamily3.MessageHandlers
Company.PolicyAdministration.ProductFamily3.Messages
Company.PolicyAdministration.ProductFamily3.Persistence
Company.PolicyAdministration.ProductFamily3.Persistence.Implementation
Company.PolicyAdministration.ProductFamily3.Etc…

Common
———————-
Company.PolicyAdministration.Common.Domain
Company.PolicyAdministration.Common.Persistence
Company.PolicyAdministration.Common.Messages
Company.PolicyAdministration.Common.ServiceHost
Company.PolicyAdministration.Common.Etc…

Do you have any thoughts on the above structure?

Once again thank you very much for your extremely valuable advice!

Best Regards,
Bill

Download via the Dr. Dobbs’ site.

Or download directly here.

Additional References:

• Ask Udi Podcast #017 – Using Autonomous Components for SLAs in SOA
• Ask Udi Podcast #020 – How to structure .NET solutions and components

Want More?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question and have him answer it on the show: podcast@UdiDahan.com.

The good folks from InfoQ interviewed me recently about NServiceBus. You can find the full interview here.

Here are some choice tidbits:

“For any developer who architects using SOA principles, on the basis for using NServiceBus over some other methodology or some other so-called enterprise service bus:”

NServiceBus makes it difficult to work in ways that will hurt your scalability. Since the asynchronous messaging patterns are brought to the fore, developers will, by default, avoid the temporal coupling so prevalent in most Web Service implementations. Other methodologies make so many other options just as easy to use that developers can make mistakes that will hurt their scalability and availability and only find out about those mistakes in production .

Another thing NServiceBus does differently is to totally isolate all workflow code from all technologies. This makes it easy to unit-test workflow classes, making it possible to iterate quickly on the development of key business processes. These portable .NET POJOs give developers the flexibility to host their workflows in whatever runtime they want.

“Having noticed NServiceBus is built with MSMQ in mind or as an option, when asked why this choice of direction: ”

The core of NServiceBus is not dependent on MSMQ. The extensibility points of NServiceBus allow you to plug in your own communication transport, subscription storage, and workflow persister. I’ve already implemented a transport over MSMQ, and another over WCF’s NetTCP binding. Developers can use those as-is or write their own. Many of the SOA products currently out there are much more tied to HTTP, so this is something of a break from the common practice.

I chose to use MSMQ since it is one of the two main Microsoft communications technologies that enables parties to communicate in a disconnected fashion (SQL Server Service Broker is the other). MSMQ has a much more accessible API in that it is directly available as a part of the .NET Framework whereas Service Broker currently isn’t. I consider disconnected communications a critical part of any SOA infrastructure since the Tenet of Service Autonomy does not allow us to assume that the service with which we wish to communicate is currently available.

Is there anything you’d like to know about NServiceBus? Drop me a comment below.

Thanks.

After reading Derek Hatchard’s post, The Art and War of Estimating and Scheduling Software, I wanted to follow up on my previous post on the topic, Don’t Trust Developers with Project Management. The problem lies with individualistic thinking.

Developers, and managers too for that matter, by and large are concerned with “productivity”. Developers want the latest tools and technologies so that they can churn out more code faster. Managers create schedules trying to get the maximum efficiency out of each one of their developers. They consider resource utilization and other terms that sound manager-ish.

Fact is, on medium to large sized projects, if you look at the studies you’ll find that developer productivity when measured as total lines of (non-blank) code of the system in production divided by the total number of developer days comes in roughly at 6. Maybe 7.

7 lines of code a day.

Let that sink in for a second.

I can hear the managers screaming already. OMFG, what were they doing all day long?! It takes, what, 10 minutes to put out 7 lines of code? An hour even, if it’s complicated recursive code and stuff. And they say they don’t like us micro-managing them?! Now we know why. It’s because they’re goofing off all day long.

Well, managers, that’s not really the way it goes. You see, you have to take into account the time it took to learn the technology, tools, frameworks, etc. Add to that the time of understanding the requirements, which is really sitting through boring meetings that don’t explain much. Finally, our poor developer actually gets to implement the requirement. Maybe run the system a couple of times, trying out the feature they implemented, and checking the code in.

Well, that’s actually the easy part. Now comes the part which kills most of the time. After a bunch of features have been developed by the team, the testers start banging away at it and find a bunch of bugs. Now the developer has to reverse-engineer some bizarre system behavior and figure out which part of the system is to blame. That involves usually some educated guessing (unless they’ve just joined the team and have been put in the bug-fixer role to “learn the system”, in which case it is thoroughly UNeducated guessing). They change some code, run the system, which looks like its been fixed, check the new code in, and close the bug.

But the bugs keep coming. And as the project progresses towards production, more and more of the developers time is spent looking through code and changing existing code, that actually writing new code.

And the larger the system, the more bugs. And I don’t mean that the number of bugs linearly increases with lines of code, or number of features. It’s probably closer to exponential. If it’s a mission critical system, the performance bugs will be taking an order of magnitude more time to fix than other bugs.

So, as you can see, getting a system into production is a team effort. It includes the developers and testers, of course, but also management, and the customer, and how they manage scope. This is kind of a “duh” statement, but we’re getting to the punch-line.

If getting a system into production involves the entire team, isn’t that obviously true for each feature too?

In which case, why are we asking just the developers to estimate the time it takes to get a feature “done”? Why are we trying so hard to measure their productivity?

I know why. It’s so we can get rid of the less productive ones and give bonuses to the more productive ones!

Back to the main issue. I don’t “trust” developer estimates because I need to see the team’s capability to put features in production. The involves all aspects, and often many team members, in some cases multiple developers going through the same code. This involves all overhead and cross team communication, sick days, etc. It’s also why I try to get multiple data points over time to understand the team’s velocity.

While I care about the quality of my developers, and testers, and everybody on my team and would like them to be able to estimate their work as best they can, I’ve got a project to put into production. And the best way I’ll know when it’ll go into production is by having data that’ll enable me to state to my management:

“Our team is finishing 20 feature-units a month, we’ve got 200 feature-units to go, so we’ll be done in around 10 months.”

If I’m busy micro-measuring each developers estimates, I won’t have the time to see the forest. By first taking a harsh look at the reality of what the team can do, I can start looking for ways to make it better. Maybe the bottleneck is between analysts and developers, maybe we’re seeing the same bugs regressing many times, but until we know where we are, we can’t run controlled experiments to see what makes us better.

Focusing on the individual developer, getting them the latest and greatest tools may be great for their morale, but it probably won’t make a bit of difference to their actual productivity.

Next time – what to do when management asks you what it’ll take to be done sooner.

Jeremy’s trying to answer some hard questions about agile. I wanted to tackle the issue of fixed-bids, since most of my clients work on those kinds of projects and I managed those projects full-time before becoming a consultant.

So, here’s the thing.

The only way to win on fixed-bid projects, is to bid low, and then rack up the change-requests. This is why people spend so much time documenting requirements, and then getting the client to sign off. It’s so they can prove that something is an actual change request, and thus they don’t have to do it. So, if the client wants to do whatever, they have to pay more money.

The problem is that it pisses off the client.

There’s another, subtler problem. It’s that clients get wise to this game, and front-load every possible requirement requesting total flexibility in everything.

This leads to another problem. We can’t bid low anymore.

Which leads to another problem. The client doesn’t have the budget to pay for the longer list of requirements.

Which leads us back to square one.

Fixed bids are a lose-lose proposition.

You see, if you bid rationally, taking into account the fact that some requirements will change, others will appear mid-way through, and so on, you’re bid will be significantly higher than the other guy who low-balled it. That means that the client will have a very hard time explaining to his management why he wants you to do the project.

So, the only way to win is for the client to realize this and game the system. This is sometimes a fine-line, possibly bordering on illegal when it comes to government contracts.

Once you have a client who understands that the fixed-bid is not in their interest, they will work collaboratively with you to get a reasonable system out the door within the given budget. There will be a lot of give-and-take but it can work. After a system goes into production successfully, it’s a lot easier to get management buy-in for the next version.

Fact is, upper management doesn’t really know all the specific requirements. So, if you don’t do them all, you’re OK, and so is your client.

In these circumstances, agile development is not only possible, but likely.

I know that it’s not really fixed-price, fixed-time, fixed-scope this way. But that’s what makes it successful 🙂

After trying a bunch of different directions it appears like I’ve found one that works. I didn’t think that I could make use of the top ServiceModel stuff, but I was wrong. Many thanks to Tomas Restrepo for making me rethink my basic assumptions.

I’ve also put in a bit of effort in setting up a bunch of configuration files allowing you to try the various transports for the example code. Please go through the Readme as it contains the detailed instructions. There are currently config files setting up non-WCF MSMQ, NetTCP, Basic HTTP, and WS Dual.

If you do go the MSMQ route, the example will run the long-running workflow code. The reason the example code won’t do that for the WCF cases is that I haven’t figured out yet the best way for a service to send a message back to itself, especially for the connected bindings. Any thoughts on this would be most appreciated.

Also, I’m open to hearing what directions you think the next version should go in. I’m currently thinking about including message transformation. Either that are doing a Multicast bus over a UDP transport.

This is a non-backwards-compatible update. In order to propertly integrate with the WCF philosophy I had to change the ITransport interface for Unicast transports. The meaning of this change is that transports are now expected to do their own threading and transaction management, while the bus object now joins the context that comes from the transport.

Download here.

I’m in the process of implementing a generic WCF transport for NServiceBus and am seriously going out of my mind. The fact that I have to do crazy reflection over generics to get around some of the channel model is ludicrous. To top it off, I now find out that there isn’t a simple generic way to send messages.

IOutputChannel exposes “void Send(Message message);” while IRequestChannel exposes “Message Request(Message message);”. Of course, they are incompatible having only an anorexic common interface IChannel (which exposes only “T GetProperty<T>() where T: class;”).

I’m doing everything in power to hide the complexity (sorry, flexibility) of WCF behind the simple IBus interface, but it looks like it’s going to bubble up in the configuration anyway.

If push comes to shove, I’ll just implement a bunch of my own transports which will specifically encapsulate each of the WCF bindings.

New release coming soon.

I’ve been receiving more and more questions about how NServiceBus fits in distributed systems and wanted to share them:

My question is about distributed topology.

The EAI-hub-and-spoke model is all about the central server. It’s useful sometimes, but there are a lot of reasons why I’m not gung-ho on using a hub as the center of the integration universe.

The ESB distributed model puts code on the endpoints. That code solves some of the messaging problems that apps face, so that apps don’t have to face them. It also solves some of the messaging problems that enterprises face, so that enterprises don’t have to face them. (I need both).

Those problems include simple coding and deployment model, pub-sub routing, reliable transport, simple transformation, and orchestration. I wonder which of these you can do in your tool, and which you are planning to do.

I’m also interested in management. How do you insure that the endpoints are correctly configured? Do you have a central configuration store? How do you propagate changes from the center to the messaging endpoints?

And here’s my response:

The important parts of NServiceBus that are independent of the distributed topology are the API and the connection to long-running workflow. This code is indeed on the endpoint. However, if you wanted to you could easily connect to something like BizTalk and do whatever you wanted there. This general idea though is to support the ESB distributed model since there’s no such things as a centralized ESB.

In terms of the capabilities you’ve mentioned, I’ve seen developers pick up the coding model in a day or two. The deployment model is just a bunch of DLLs you deploy with each endpoint. Dependency Injection is supported by www.SpringFramework.net but you can replace that with something else easily as another implementation of the ObjectBuilder interfaces.

Currently pub/sub routing is supported over regular point-to-point transports in a transport agnostic way. You also have the ability to have subscriptions be persisted so that even if a server restarts (and clients don’t, and can’t know about that) all the subscriptions will be remembered.

The reliable transport that is currently supported is MSMQ, with the option of defining per-message type if you want durable messaging (using the [Recoverable] attribute).

In terms of orchestration you get a nice model for long-running workflow that gets kicked off by messages decorated with the [StartsWorkflow] attribute, and messages that implement the IWorkflowMessage interface get automatically routed to the persistent workflow instance. You have the ability to change the storage of workflow instances easily as well. Workflows are simple classes which are easily unit-testable in that they expose a “void Handle(T message);” method for every message type (T) that is involved in the workflow.

I haven’t done anything in terms of simple transformation yet but am currently looking for the right place in the message processing pipeline to put it. I also haven’t done anything yet in terms of management.

What is currently being done management-wise on the projects that use it are the commercial options for managing configuration files in distributed environments coupled with the regular ability to restart windows services and IIS applications. I haven’t seen anything lacking in that solution yet.

If you have any questions, please don’t hesitate to send them my way – NServiceBus@UdiDahan.com.

I’ve been asked if I have a document describing the architecture of NServiceBus, so since I don’t have one yet, this post will be the start.

First of all, there were two major forces that drove me to write NServiceBus. First of all I wanted to formalize the way Service Layer classes were written when using asynchronous messaging (regardless of Web Services). And second I wanted to formalize a communications API which supported pub/sub so that I could freely move between transports – whether they inherently supported pub/sub or not (the fact that Indigo did not falls under this).

From there it was filling in the details. Until finally I made the step to formalize the way long-running workflow connects to asynchronous messaging. That’s the third pillar as far as I’m concerned about ESBs. Using MSMQ I do enable Shared Subscriptions like Sonic does.

I try to stay away from the vendor side of ESB definitions and just deal with what capabilities such a library should provide. As such, it is inherently distributed – running on each server/client.