The people side of performance

Those of you that read my blog know that I tend to primarily address technical topics when it comes to performance. The latest issue of Communications from the ACM (10/2009 vol 52 No 10) has an article called "The Business of Software Contagious Craziness, Spreading Sanity" by Philip G Armour. I was intrigued with some of the points Philip brought up in his article. Particularly this tidbit

"... having a strong conviction that something cannot be done is usually a self-fulfilling prophecy. If people are convinced that something is not achievable, then they usually won't achieve it - if we argue for our limitations, we get to keep them."

Wow! Nail on the head. So how do we convince an organization that something that is currently perceived as impossible actually is quite achievable?

First off, people have be encouraged to think outside the box. George Patton is quoted as saying "If everyone is thinking alike, someone isn't thinking." Coming from the battlefield I think we can understand why the General said this. He needed his commanders to not only visualize the current plan of operations but also what alternatives and/or competing strategies were available. One of my friends, Keys Botzum who also happens to be an IBM STSM, once put it this way. "The patient is dying on the table. What can we do, right now, to keep him from dying." It is a different perspective for techies to mull over.

Likewise, a technical team can not be lead by consensus. Technical teams have be be lead by dictatorship. Someone, at the top, needs to make technical decisions and take responsibility for those decisions. Right or wrong someone, like General Patton, has to tell the troops which way to charge. He can take input from his Lieutenants however the decision is ultimately his and his alone to ensure it is the correct decision. Which is why the other side, led by mad men who knew nothing about how to fight a war, lost. The General knew what he was doing and knew how to lead his people to victory.

Which brings us to our next topic. In order to be able to lead a technical team the leader has to be technical. I understand management's philosophy that they are managing people and not the technology. The problem is that technology, by its very nature, is complex. This is simply unavoidable. We're not managing a bunch of lawyers (sorry to pick on lawyers, you're used to it by now, as they are a good example). What lawyers do is not very hard to comprehend. They sue people. They defend people. They argue in court for their client. They draw up contracts, etc. None of this is rocket science. Start running multiple servers in parallel handling thousands of transactions per second and that organization is starting to approach NASA-like technical complexity issues. In order for a technical team to succeed they must be lead by a strong technical person. It is unreasonable to expect someone who does not comprehend the subtleties of technical subjects such as "verbose GC" or "JDBC deadlocks" to be capable of making a technical decision much less the correct one.

In conclusion, and back to the "self-fulling prophecy" and "that something is not achievable" that started this line of thinking for me. There are two ways to avoid failure. One is to have a strong technical leader who can take the reins as the technical General who makes the decisions. The other is for executive management to support the technical leaders. This means not only in terms of vocal agreement but also in terms of financial dollars to the project, making people available to do the tasks and any other strings that need to be pulled in order to make things happen.

I'll have more to say on this subject later as this is about all I can muster in the limited time I had for this "lunch & learn" session.

Managing the application threads

Threads and multi-threaded programming is not a trivial subject. But some application developers find themselves having to start their own threads as part of their application.

I'll start this post off saying that using unmanaged threads is never a good thing. When using threads developers should be looking at how to manage all their threads in their application. Never exit the application without terminating all of your threads. Why would we need to do this?

An interesting problem is when an application is stopped (not the JVM/app-server but just the application), a new version is deployed, but the restart fails because there are still threads hanging around from the previous version of the applications and older versions of the same classes. I never would have imagined this problem occurring but I can see that if the application lost track of its threads that it would not be able to close them when it shut down.

Java provides the capability to manage all the application threads by using thread groups. When creating a thread and assigning the application's thread to a thread group all the application has to do is capture the "application stop event" and use the stop method of the thread group thereby stopping all the threads in the thread group.

The beauty of this solution is that the application can reliably manage all of the threads it creates though a single interface. It is also portable which means your application will work with any application server.

High availability through parallel cells

High availability sites have unique requirements that are not easy to solve. However, one solution does revolve around using multiple cells in production. This allows for easily taking a cell out of rotation to make changes. Then when the cell goes live and if the changes didn't work it is just as easy to take the cell out of rotation again. I've used this strategy at many, many 24 x 7 sites.

Here is the link to my latest article on multiple cells.

The only problem that arises is if things like database schemas change in the backend which also means application code changes. Those kinds of changes require more work and possibly parallel schemas until the transition is complete.