Some common mistakes

Sending the right files
I spent a frustrating day yesterday reading log files from a server that I was told had stopped running (i.e. the process was gone). I spent a long time reading the logs looking for an indication of a problem and couldn't find one. On examining the IP addresses in the logs it was clear they had sent me the logs from the wrong server. Sysadmins, make sure you send the right log files. Once I had the right log files I had the answer within the hour and a fix to go forward with.

Verbose GC
Too many people run with verbose GC turned off. Unfortunately the word verbose is over loaded and in this case verbose does not mean verbose in the Unix sense. Verbose GC is actually not that verbose and has a heck of a lot of great information in there. Run with the ST_VERIFY option if you are on pre-Java5 JVMs (i.e. earlier than WAS v6.1)

Stay current with maintenance
Time and time again I go in to debug a problem and what do I find? The problem is "known" and a fix has been available for some time. Just because things are working today doesn't mean that they won't break in the future. Yeah, I also understand the "if it ain't broke don't fix it" mentality and application servers are one place I don't like to play that card. Run the maintenance through all your test environments (including your performance test environment... you do have one, right?) and things should be okay in production. But don't expect to run on a 5 year old JVM and not run into problems one day...

I'm sure I'll think of more of my pet peeves so I'll add them later. In the meantime, if you're bored and need something to read see my paper on large topologies.

HTTP tuning

I often have to modify the performance of the HTTP side of the conversation and here is a link to a decent basics:

Best Practices for Speeding Up Your Web Site

I also like the fact they used IBM's Page Detailer http://alphaworks.ibm.com/tech/pagedetailer which is an awesome application my colleague performance whiz Phil Theiller showed me. It is really awesome for getting a good visual feel for the HTTP side of things.

Lack of Performance Testing (load and stress)

One thing I can not stress thoroughly enough is the necessity to conduct performance testing. I've already written an article on this subject (click here) and there is no reason not to do it. In fact, the #1 reason most critsits happen is either the complete lack of performance testing or very poorly executed testing. This is what I refer to as self-inflicted pain because, from my point of view, if you make the decision not to thoroughly test then you have accepted the reality that you will have a production outage and it will be difficult to resolve.

In addition, you should be monitoring your application using an appropriate tool like IBM Tivoli's ITCAMfWAS (IT Composite Application Manager for WebSphere Application Server). v6.1 was just released and I got a chance to play with it at a critsit over the past couple of weeks and it is a good improvement over v6.0 addressing some of the usability issues I had. It is tools like this that help me resolve a critsit in a matter of hours instead of days or weeks.

Day One SHAREABLE JDBC connection setting

I do not know why I didn't start a blog on this topic earlier! This is what I do day in and day out so having an outlet where I can describe and discuss various scenarios is going to be helpful to me.

So, for our first topic we are going to talk about a common malfeasance in production performance that typically negatively affects any application that runs in the WebSphere Application Server environment. Okay, maybe not all that common but I've seen it three times now and obviously someone needs to start paying attention.

Inside the application EAR file there are deployment descriptors for the JDBC connection pool that refer to SHAREABLE and UNSHAREABLE connections. While there is plenty of documentation on this particular deployment descriptor there seems to be little understanding of how this particular setting works. BTW the default setting is SHAREABLE which is where most problems seem to stem from.

Now, what I'm writing here is not gospel so you will need to understand your application and how it uses JDBC connections particularly if there are transactional UOW involved. However, for the other 90% of you that are doing simple SQL queries and a few inserts here and there you will want to pay close attention... change the descriptor to UNSHAREABLE and run your performance test suite (you do have a performance load test, right?) and notice the change in behaviour of the JDBC connection pools. If you did this right you should see fewer connections being used.

This is because the default setting of SHAREABLE does not return the connection to the pool when you call close(). In fact, that connection is held with the thread until the thread finishes processing the request. This is an optimization that some people may need but the majority do not.

So the next question people always ask me is "why is this the default setting?" Well, heck, someone had to decide on what to use as a default and they picked one. But just because something is a default setting does not necessarily mean it is the right setting for your application. I think the JVM defaults to 128M for a maximum and you can pretty sure that probably is not the right maximum for 99% of the J2EE-based applications out there!