After going through several articles and posts and comments about images, I’ve decided to devote some space to the issue here. The treatment will be done at system-wide levels, meaning with an eye on all the *ilities. The debate is between saving images to the/a database vs saving them to disk. My choice is disk. Here’s why.

First of all, some history. Once upon a time, the way all web apps handled images was with the <img> tag using the “src” attribute to point to the relative location of the image on the server’s disk.

Over time, the idea of keeping all entity information together spread, and we began saving images to the database, and then returning them to the client by streaming them over HTTP. We began using the “src” attribute of the <img> tag to point to .aspx pages or .ashx handlers which would return the image to us. Thumbnails and zoomed in images were created on the fly by these techniques and everything worked. But not for long. These techniques begin to break down when the load increases.

What are the problems that occur when saving images to the database ? Well, for starters, if you save the image in the same table as you store the rest of the entity’s information ( for instance your employees table ) you will most likely overrun the page size for the row causing it to be split over multiple pages. This adversely affects performance. ( Note: if you do decide to go this route, I suggest creating a separate image table where you have an Id and the image itself. Then, from your entity table – employees – use a foreign key to the image table. )

Although the previous problem adversely affects the performance of the database engine, this next problem is more of a gotcha. When saving images to disk on the server, the client is able to cache them by itself to its own disk. This reduces the web server load over time as it has less bytes to serve. When images are referenced by “GetImage.ashx?EmployeeId=12”, although you can cache the bytes retrieved from the database in memory on the web server, the client doesn’t cache them to its disk. This further reduces the scalability of the system.

Finally, consider the issue of multiple sizes of images – thumbnails, zooms, etc. Although these can be dynamically generated quite comfortably with “GetImage.ashx?EmployeeId=12&Width=100&Height=100”, this increases ( per request ) the number of CPU intensive operations needed. A prefered alternative is to automatically generate all the image sizes required once, save them to disk, and serve them like any regular image.

Personally, I use an Imaging service for all of these needs – saving the original image to disk, generating thumbnails, getting the path of an image for a given entity ( eg. employee ) and a given size ( eg. zoom ).

Finally consider the case where you have a large volume of images to serve. Better to have a single server, with ultra-fast disk access + Raid, which serves all images. This server, when separated from the database server, would greatly increase the manageability of the server farm and make diagnosing performance problems much easier.

To conclude, although an interesting capability, saving images to the database begins to pale when compared to alternatives under load. Don’t get me wrong, for small systems it may be the simplist 🙂 thing to do, but, time and time again, that small system you built 2 years ago is required to scale and support much more than you originally intended. And, its really not that hard to work the “old fashioned” way. ( Note: run-time generated graphs based on data should not be moved to disk. )

I’ve been contemplating the move to SOA style development for some time now, but recently I’ve taken the plunge. I’m talking, of course, about giving up OO as a way of life. Not everything is an object.

I have a lot to say on the subject, but my main focus will be on the impact SOA has for developing a system. That’s right. One system. Not how to connect different systems using XML Web Services.

On that note, I’d just like to debunk a myth I ran into recently a la 15 Seconds : Realizing a Service-Oriented Architecture with .NET. The following image appears to be “common knowledge” when talking about SOA. Anyone see the problem ?

Hint: its between the Business Objects and the Data/Persistence.

The problem is that persistence is ( or at least should be ) a service. The business objects themselves should not be aware of their persistence. Rather, the services above should use the persistence service in order to perform the work needed.

Instead of: myCustomer.Update();

Should be: Persistence.Update( myCustomer );

Why is this important ? Because it contributes to a separation of concerns ( see Agile Management – http://www.agilemanagement.net/Articles/Weblog/Separationofconcerns.html ). Obviously, putting the knowledge of persistence ( or, for that matter, anything else ) into business objects creates bloated objects that need to know everything about what can be done with them. I’m sure that this will just further inflame the great O/R mapping debate ( see “Futility of Object-Relational Technologies” and “GENerative Object Mapping Layer” ) but it is still true.

Coming up next:

How to do persistence with SOA.

Several times in the career of a “programmer” one considers opening up shop and becoming his own boss – becoming a Consultant ( the capital C stands for the extra money consultants demand ). Brady brings up some interesting issues budding consultants often encounter. Robert offers some really great advice from his own consulting process. I especially appreciate the value of using escrow services. (Update: Robert adds some more advice here ) I’d like to share what little experience I’ve gained over the years in the hope that some other consultant just starting out may save himself some grief.

From my experience, the main skills area that needs improvement in programmer-turned-consultants is project management. This area deals with everything around the actual product/system developed. Development really isn’t the problem. Bottom line – first and foremost, READ ! Anything and everything about project management, contracts, RFPs, and all other things relevant in some way to the project. The hard part is always implementing all the great accumulated wisdom out there.

The most critical thing to remember is: It Takes Time.

Not pure programming, in the zone, 100% productivity time. Time to develop relationships with stakeholders. Time to go through the RFP/RFI process. Time to negotiate. Time to travel ( a lot of people when starting out forget to take this into account when budgeting – it IS time spent on the project, and should be measured – at the very least). Time to meet with users. Time to learn the terrain. Time to reconcile differing views. Time to play politics. … Time.

Obviously there isn’t enough room here to go over all the skills and techniques of project management and consulting, however, I would be remiss if I didn’t come full circle from time to money.

First of all, unless you have a signed contract, signed detailed project description document, etc… no project will be done in 2 months. Once you have all those things, AND there are NO mid-project changes, then you have a case of “contract programming” mentioned in the comments of Brady’s entry, which has a chance of being done in 2 months.

However, there are end-project activities which have to take place, installation, training, support ( yes, even if you didn’t include it in your offer, if you don’t give some support for the bugs that will doubtlessly come up, the project will be labeled a failure. ), meeting with the major stakeholders to see that they’re happy, … you might as well not have done the project.

So, in essence, once you’re a consultant, work != programming. In the vast majority of cases non-programming.Time > programming.Time. Its this mindset that has to take hold. This particularly becomes difficult when a potential client comes to you with a project that sounds exactly like something you just did. It happened to me recently. A client asked for a system that I just finished rolling out at a different client, and I thought: Easy money. Well, the politics there were like nothing I’d ever seen. The project took even longer than at the first client, even though the code had already been written !

Bottom line: Everything is a project, and must be treated as such.

Now, getting to money. Money is important, and so is when you get it ( see Robert first post about milestones ) and making sure that you get it ( Escrow a la Robert ). And, don’t forget taxes. For one-person shops that aren’t overflowing with projects, deferring a payment to a new fiscal year can make a big difference to your net income, and may buy you a favor with the client if you play your cards right.

Of course, no discussion about money and consulting would be complete without raising the issue of hourly vs flat-fee pricing. Most clients prefer fixed price offers, since they fit with their yearly budget planning.

Many consultants I’ve met use hourly fees, but billable hours is a fickle measure that varies from project to project. Personally, I have certain issues with the hourly pricing model – note that I’m referring to the invoice the client receives. When charging hourly, the client will obviously want to know what you did on an hourly basis. Try explaining to the client to pay you $X/hour for having lunch with the head of computing services to make sure that he was pleased with the effect the project had on his department.

Fixed price offers allow a consultant to roll up many project expenses that are often difficult to collect from clients in an hourly model. For those who are skilled in project management, this model often works well. However, the risk involved in under-estimating scope, or time required, may adversely impact the bottom line.

Recently, after reading this great article on Value-Based fees, I’ve had great success in this model. In a nutshell, you, in cooperation with the client, assign value to each deliverable, and price it. This model handles scope creep very elegantly. For lack of space, I suggest you go read the entire article. Moreso, the “Million Dollar Consultant” has a whole page of tips for those going the consulting route. Highly recommended.

To sum up, to succeed in consulting one has to treat it as a whole different career path to be learned and lived. Think “Career Calculus” by Eric Sink. More importantly, consultants are PEOPLE people, not computer people. Soft skills rule.

Prototypes. They’re everywhere. Sometimes they grow into full-fledged systems. Sometimes they’re thrown away. But they can be considered a project in their own right. I’ve decided to write this entry after reading Fabrice’s thoughts on the same subject. Before getting into my own thoughts on the matter, I’d like to quickly sum up the main points I found there, including the comments posted.

The main issue originally revolved around web apps, and whether HTML or ASP.NET should be used for prototypes. Powerpoint, Visio, Paper – yes that stuff that comes out of printers, Photoshop, Illustrator, and Denim were all brought up. Arguments for and against were made. Before I pick a side, I thought I’d return to the basics.

A prototype, by definition, should be thrown away. The lessons learned, obviously, should not.

I bring this point up as it is the most basic. When thinking about prototypes, the last thing that should concern us is if we can “reuse” pieces of it for the actual system. Now, if the point of the prototype is to learn, and learning is most facilitated by short feedback loops (tons of research back this up, think pavlov), we should be looking for the “technology” that allows for the quickest feedback.

What do I mean by feedback ? What the client thinks about what we’re doing. And what’s the fastest form of “do something, get feedback” ? Talking. What we need is a way to actually DO the prototype WHILE talking to the client. This pretty much reduces the number of contending technologies down to 1. Paper.

Paper prototyping, when done well, brings to light SO many issues and unspoken assumptions in SO little time, that, from my experience, it just doesn’t make sense to go any other route. There is so much information out there on paper prototyping that I can’t even begin to list it here. Just google it.

So, how can paper prototyping be done well ? Well, it requires preparation, and, obviously, lots of paper. Plan for about a 1 hour sessions at a time with the client, no more. These sessions are VERY tiring. If possible, try to get someone to write down everything that came up during the session, and what resolutions were reached. This person should not be involved in the session beyond being an observer.

Now, let’s consider the alternative. Using a computer + some software, be it Visio, Photoshop, whatever. The problem with these approaches is that we get to focused on what we’re building, and lose focus of the lessons we should be learning. These approaches obviously increase the time between a decision we make about the system, and getting the client’s reaction. It therefore also increases the time we waste when we get a decision wrong.

The Agile world has already come to this conclusion and makes every attempt to get a real, live customer/client/user on-site for the development of the actual system. When building actual systems, obviously, paper can’t be used. But where it can, and it’s most appropriate, it should.

To sum up, when a prototype is done, we should know more about the real system that we’re building than when we started. Short feedback loops with our clients is what gives us the most information. And paper is the medium most suited to short feedback loops when dealing with prototypes.

Update:

Six Signs That You Should Use Paper Prototyping from Java.Net

Sign 1. There are many different ideas about the design

Sign 2. You find yourself defending a particular design

Sign 3. There are parts of the design you’re unsure about

Sign 4. You’re changing the way that the users perform a task

Sign 5. The concepts or terminology are new to the users

Sign 6. You’re feeling uncreative

There’s a building buzz about SOA. I don’t think that anyone can argue with that. Also, I believe that for large systems, SOA will become the new paradigm. Don’t take my word for it, the PAG group at Microsoft have recently began rolling out ShadowFax, the still emerging definition of what Microsoft sees as SOA’s future.

Now, as a developer, I am pro TDD, using it constantly and consistently. I have been thinking a lot about XP as a way of developing systems in teams. The success stories have been increasing over time, and XP appears to be a valid development methodology in its own right. Not only XP of course, the entire Agile camp ( XP, Scrum, DSDM, etc ) has been raising some serious concern in “heavy” development shops.

What interests me is the mesh/meld of Architecture ( yes, with a capital A ) and agile methods. It would appear that the two, although not entirely opposed, don’t quite fit. Agile methods allow the system’s architecture to unfold as the system is built, keeping it as supple as possible. The architecture camp, of which SOA is a member, define the system’s architecture up front. Of course, small intra-service development efforts could occur before or in parallel to the architecture definition process, however, their effect on it would be minimal at best.

From a historic perspective, I think that where SOA comes in, Agile methods have yet to be introduced, or, if they have been already introduced, the data as to the success of agile over non-agile are inconclusive at best. Very Large Systems ( once again, capitalized ), like those developed for financial institutions, the military, and government have seen very little agile action at the project-wide scale. These systems have multi-year development+deployment schedules. I have recently done a stint at one such project. Waterfall, although not called by name due to recent bad press, is still used, but under the guise of tailoring some other development process.

On these large-scale development projects is where SOA will have the short-term largest impact. This is for the simple reason that project architectures are all different, but there’s nearly always only one development methodology for the company. The entrance barriers to SOA are much, much lower than any agile method.

I would be very interested in hearing any accounts of the SOA-Agile mix. Any thoughts would definitely be appreciated.

Also, “small-scale” SOA sounds like an interesting idea. How relevant is SOA to not-so-large systems ? As a standalone developer, I’d like to know what improvements SOA can bring me over OOA and other alternatives. I’m always looking for ways to be more productive and increase quality at the same time. TDD has recently done exactly that. Some of XP’s practices have also contributed. SOA looks promising. Its the integration of these behaviours that interests me today.

Tell me what you think !

I haven’t yet found a mistake of mine that I want to publish, but it just so happens that I came across someone else’s ( luckily ).

On Udi Dahan’s Projects you can find information about the projects that I’ve currently got underway – 5 ( +1 that I’ll have to take on later ).

On one of the projects, Accounting System, I am charged with taking a project that’s finished development ~95% and taking it to production. The developer took me through the system, and I realized that this was not the architecture that I had envisioned when told: “This is a good system. Well architected, stored procedures, and C# code that calls it. ASP.NET pages using code-behind. Its practically finished.”

You can see what kind of monster I’ve inherited here.

To all you DBAs out there who want to develop systems, I tell you this: Do NOT put all the system into the database just because you can ! Even the dreaded “3-tier architecture” is better than that.

Just so the rest of you get an idea of what I’m talking about, think about putting all possible logic into the DB. Calling a stored procedure is all you ever do from the UI besides basic GUI event handling stuff. The SP then does everything – checks business rules, sends email, calls webservices, performs calculations – you name it.

And WHY did they do this ?! So that they could change the functionality of the system without recompiling it ! Let’s see, toss out all the *abilities ( maintainability and all the others ) , get ability to change system without recompile. Hmm… That’s like trading diamonds for glass.

Now, here’s the best part: For the project to be profitable, I’ll have to work by the same conventions ! ugghh…

Final thought: “It works” isn’t an excuse.

Well, I guess that my mistake in this whole process was assuming. Assuming that what I consider well architected, and what someone else considers well architected are two different things. Or, more generally, miscommunication. Even though I was told what to expect “Well architected, stored procedures, and C# code that calls it…”, I didn’t verify what this exactly meant. And even worse – I gave an estimate based on my assumption.

Lesson to be learned ( for me – and I’m only writing this so that I can berate myself more harshly the NEXT time it’ll happen ): Actively search for the hidden assumptions that I make when listening to someone. Make them explicit. Verify them with the speaker.

Final note: although I’ve heard this piece of advice many times before, and, like most things, is just common sense, it wasn’t REAL until it happened to me.

Once upon a time I thought I had something to say. When I said it, it was received quite well.

Today, I have read and heard what so many others have to say, and I feel that everything’s been said. Not only that, but they say it so much better than I ever could. Is there any point in saying something that’s already been said elsewhere ? What if you haven’t seen it elsewhere yet ? Well, in that case I’m still pretty sure that I’ll just be re-hashing ( at best ) what my betters have already said.

So, what’s left to be said ?

Only one thing.

My mistakes. Maybe from my mistakes, other can spare themselves. This, IMHO, can be my greatest contribution to the community.

I thus set out to let the world know of my screw-ups, and what I learned from them ( since, as many already know, you don’t learn much at all from success ). This will be my mission.

Update:

I just found this article on Fool.com about the 12 simple secrets of Microsoft management. Point 4 states: “Require Failure” as follows:

“At most companies, to succeed is good, but to fail is unacceptable. This type of policy means that, as a risk/reward scenario, the risk of failure vastly exceeds the reward of success. Thus, most companies suffer from a workforce that pursues a course of failure avoidance. In contrast, at Microsoft, failure is expected, and even required because risking failure is the only way to push the envelope. As a result, Microsofties relentlessly pursue success without fear of failure. And if they fail, they understand that the key is to fail quickly and not waste time. “

This is great. It turns out that I’ve been doing things the Microsoft way all along.

Things have been quite frantic recently ( for a glimpse of what’s going on see Udi Dahan’s Projects under the categories ) so I haven’t had much time for blogging. I know that everyone gets bogged down in work now and again, and apologizes for not blogging, but I’m actually quite happy that business is going so well that a large portion of my time can now be billable. If this keeps up, I’ll have to reconsider non-work blogging.

So, although I doubt that this will be my “last” entry, I will probably be going into a dry spell.

Recently I’ve started thinking about other potential uses for blogging and rss in general. One thought that came to my mind is the use of a blog for managing projects. Before I get into the managing projects thing, I guess that I should give a little background about myself. I am a standalone programmer / consultant. This really means that I am a team of one. I meet with clients, market my services, sign the contracts, meet with the users, dig for requirements, decide on architecture, design the UI, code the system ( test first ), deploy, manage user expectations, etc… Not only that, but I do this on several projects at the same time. This is what I mean by managing projects. I need a way to be most effective at doing all of these things. Just think of the possibilities if all of the stakeholders on a project had an rss aggregator. Communicating status on an ongoing basis would be simple. Mocking up a UI and getting the users comments would be simple. If you could get wiki technology integrated, you could work on all sorts of requirements docs/xp stories/whatever simply. To make a long story short, its all about effective asynchronous communication. Anyway, after all of these thoughts I came to a decision. Just do it. Try it. What’s the worst thing that could happen ? It’ll flop, and I’ll stop it pretty quick. On the other hand, maybe it really could be good for something, even a subset of the things listed above. Maybe other things that I didn’t think of. Maybe the project stakeholders will come up with other ways to use this technology. So… I am officially anouncing the opening of Udi Dahan’s Projects, the blog that will change the way we work – well, the way I work. The first experiment of its kind – that I know of. Tell me what you think ! How can blogging be used to manage projects, or, more generally, to improve the way projects are run ? Any ideas, crazy ones are exactly what I’m looking for, on other uses for this technology ? Am I totally wrong and going on the path to destruction ?

I’ve been reading a lot from the agile management blog lately, and I came upon this entry about implementing business rules with aspects. If you haven’t yet read this previous entry about the division of the system architecture not only along system lines, but also along “risk of change” lines, then I suggest you read it first.

First of all, I would like both developers and managers to read the previous entry because of its profound meaning. We CAN and should separate different parts of the system having differing levels of risk. Once again, “3-tier architectures” don’t address these issues. So, if we are dividing the architecture along risk lines as well, then obviously the UI would be the riskiest – having the highest probability to change – over the life of the project.

However, business rules are often considered quite stable. Why ? Because their time-line of change is often stretched out over many project life-times – rather obvious really, business rules deal with the business, not the project. During development of the project we are often given the requirements in such a way that it is difficult to know what is domain knowledge and what is a business rule. It takes effort to separate out what are the business rules – what has changed in the past, and what will change in the future.

For example, in an academic project I performed some time ago, I was given a set of requirements including: 1. A student can register to a given course once in a semester. 2. A student can register to a project in a semester. Which is the business rule, which is the domain ? In this case (1) is the domain, and (2) is the rule – “a project” meaning “only one project” as I later found out. This rule changed sometime after my second alpha to “A student can register to only one project alone in a semester, but several projects in the same semester as long as they have a partner for each of those projects.”

As it is apparent, I, as a developer, have to take into account these risk/change factors and change my architecture accordingly. Ever since that project I have always created a business rules layer separate from the infamous “BL” ( Business Logic ).

Now, getting to the issue of implemeting using aspects. I am a big proponent of AOP, however I often find alternative implementations to be more desirable. Most of the examples given for the use of AOP including logging, transactions, security and others that can be made part of a framework, as .Net has done in many cases. Many have pushed Java’s superiority because of AspectJ, and although .Net doesn’t have an Aspect.Net yet ( although various developments are under way ), I have yet to miss it. Clemens Vasters has done some truly incredible stuff in the use of attributes in .Net for implementing AOP stuff. I’ve always thought that attributes should be used that way. I find that there is one basic flaw in the conclusion to using aspects for connecting business rules to systems. The entire white paper is here. ( Yes, its a pdf unfortunately – only good for print really, Jakob Nielsen thinks so too. ) The basic premise is that I’m building an entire system in an OO manner, which I’ve stopped doing some time ago for these reasons. Clemens has great insights on this as well, see them here.

When using an SOA, your UI, or any other system needing services for that matter, will be sending messages to the guts of your system – the “BL” for you hardcore 3-tier-guys. Let’s call the thing receiving messages in this case the Gateway. All the gateway does is receive messages ( like “Register student number 12 to course number 15 in semester 21” ) and pass them on to the appropriate handler. The word “handler” hear is used like in the term “event handler” for a reason: The receipt of a message is an event. In the white-paper the authors refer to these events as a problem that has to be dealt with. Why ? Because when working in an object oriented fashion, you would have to intercept the call to: new Student(12).RegisterToProject(93); in order to handle the event, ie check/activate a business rule. Aspects are great for this sort of thing. However, when working in a service-oriented fashion, you would send a message of type “RegisterStudentToProject” with the parameters StudentID and ProjectID as above. No need to intercept any call since it has to first go through the gateway. The gateway would then pass the message to the business rules engine which would then find and activate the appropriate rules before and after the actual call to register the student.

The rules engine does something like this:

If ( ActivateBusinessRulesForMessageAndReturnTrueIfCanMakeCall(myMessage) )
{
MakeCallForMessage(myMessage);
ActivateBusinessRulesAfterMessage(myMessage);
}

The business rules themeselves are implemented in a separate layer than the engine. The mapping between rules and messages is also done in a layer separate from both the engine and the rules. Once we have a layer for each of these, we have architecturally separated the parts that change more often in the system from the rest of it. One can also move to a more dynamic model. One in which you define a language for defining rules, and the mapping to messages as well. Thus, changes could be made by changing a configuration file instead of recompiling any part of the system.

Note that when you have lots of rules and the order for activating them matters, you should move to a commercial rule engine instead of implementing your own. You’ll see that performance becomes an issue as the number of rules increases.

I hope that I’ve managed to introduce yet another strength of the SOA over the pure OO paradigm. Tell me what you think ! Where does the SOA fall short ? Where does the OOA beat the SOA ? Am I full of it ?