Alexandre Polozoff on WebSphere Performance

Backup Strategies

2011-03-30T07:03:00.000-07:00

I was having a discussion recently with one of my colleagues around server backups. He likened it to the spare tire of the car. Yes, you can drive around without the spare tire but does anyone want to do that for long stretches of time? Probably not.

Servers need to be backed up. Simply if a server completely tipped over it can be duplicated, rebuilt and put back into service.

tprof switches

2011-02-07T12:32:00.001-08:00

I wanted to capture this before I forgot about it ... tprof switches

Cognos

2010-11-30T06:10:00.000-08:00

Looking for information about Cognos? I was. My colleague in the UK, Richard Collins, was able to point me to some informational links about Cognos.

For full details on this product, see its home page here: http://www-01.ibm.com/software/analytics/cognos/business-intelligence/

However, your most useful starting point is probably going to be: http://www-01.ibm.com/support/docview.wss?uid=swg27014432

...and in particular, this one: http://www-01.ibm.com/support/docview.wss?uid=swg27019126

global transaction sharing

2010-11-09T17:49:00.000-08:00

In the developers resource references there is an attribute for global transaction sharing.The value should be Shareable if the application uses global transactions. But if one is using LTC then one does not need to share the connection. Change the res-sharing-scope to Unshareable as noted above. This eliminates a lot of contention for connection pool threads. For more details on LTC see the link above to understand how it works.

The importance of dynamic circuit breakers

2010-09-24T05:24:00.000-07:00

It is rare for anyone to provide details behind the root cause of a production outage. Facebook put out a report about an outage they had. If you're into troubleshooting and problem determination it is an interesting read. It sounds like they could turn off the particular function but had to completely restart the environment to do it. This is why it is important to have circuit breakers that can be activated dynamically.

One also wonders what infrastructural changes could be made in the environment to help? It sounds like the application logic continued to retry requests. This is why I'm not a fan of applications automatically retrying requests because when failure occurs the retries can quickly overwhelm the back-ends. A firewall could have at least help shut off the pipe to the database. Though the consequences to the application would have been no different and would still have required a restart since there seemed to be no way to dynamically shut off that particular function.

Certainly the error logic sounded confusing at best. And error paths through code are the ones least frequently tested so they tend to fail magnificently in production

grep Exception SystemOut.log | wc -l

2010-09-14T15:46:00.000-07:00

I'm reminded today that after performance tests a simple check exists especially when adding more JVMs to a cluster. Count the number of exceptions in the log files. Of course, clear the logs before running the test. If the counts are not all roughly the same (or significantly skewed from the other app servers) then it is clear there are issues with that JVM that need to be checked. Sometimes it is configuration or a misplaced JAR file.

Been a busy 2010

2010-06-08T04:33:00.000-07:00

I know I haven't kept up with the blog this year. While the blog post I'm linking to today makes no mention of performance it has everything to do with performance. Maybe one day I'll get a chance to sit down and explain my thinking.

Catching up on closing up 2009

2009-12-24T06:53:00.000-08:00

It has been a busy few weeks for me which is one reason I haven't posted in a while. Some things I've been thinking about lately revolve around high availability, switches and logical partitioning of applications and application servers. I'll try to post more on these subjects over the next few weeks.

Hope you all have a great holiday season and a Happy New Year. I'll see you on the other side of 2010!

The people side of performance

2009-11-10T09:18:00.001-08:00

Those of you that read my blog know that I tend to primarily address technical topics when it comes to performance. The latest issue of Communications from the ACM (10/2009 vol 52 No 10) has an article called "The Business of Software Contagious Craziness, Spreading Sanity" by Philip G Armour. I was intrigued with some of the points Philip brought up in his article. Particularly this tidbit

"... having a strong conviction that something cannot be done is usually a self-fulfilling prophecy. If people are convinced that something is not achievable, then they usually won't achieve it - if we argue for our limitations, we get to keep them."

Wow! Nail on the head. So how do we convince an organization that something that is currently perceived as impossible actually is quite achievable?

First off, people have be encouraged to think outside the box. George Patton is quoted as saying "If everyone is thinking alike, someone isn't thinking." Coming from the battlefield I think we can understand why the General said this. He needed his commanders to not only visualize the current plan of operations but also what alternatives and/or competing strategies were available. One of my friends, Keys Botzum who also happens to be an IBM STSM, once put it this way. "The patient is dying on the table. What can we do, right now, to keep him from dying." It is a different perspective for techies to mull over.

Likewise, a technical team can not be lead by consensus. Technical teams have be be lead by dictatorship. Someone, at the top, needs to make technical decisions and take responsibility for those decisions. Right or wrong someone, like General Patton, has to tell the troops which way to charge. He can take input from his Lieutenants however the decision is ultimately his and his alone to ensure it is the correct decision. Which is why the other side, led by mad men who knew nothing about how to fight a war, lost. The General knew what he was doing and knew how to lead his people to victory.

Which brings us to our next topic. In order to be able to lead a technical team the leader has to be technical. I understand management's philosophy that they are managing people and not the technology. The problem is that technology, by its very nature, is complex. This is simply unavoidable. We're not managing a bunch of lawyers (sorry to pick on lawyers, you're used to it by now, as they are a good example). What lawyers do is not very hard to comprehend. They sue people. They defend people. They argue in court for their client. They draw up contracts, etc. None of this is rocket science. Start running multiple servers in parallel handling thousands of transactions per second and that organization is starting to approach NASA-like technical complexity issues. In order for a technical team to succeed they must be lead by a strong technical person. It is unreasonable to expect someone who does not comprehend the subtleties of technical subjects such as "verbose GC" or "JDBC deadlocks" to be capable of making a technical decision much less the correct one.

In conclusion, and back to the "self-fulling prophecy" and "that something is not achievable" that started this line of thinking for me. There are two ways to avoid failure. One is to have a strong technical leader who can take the reins as the technical General who makes the decisions. The other is for executive management to support the technical leaders. This means not only in terms of vocal agreement but also in terms of financial dollars to the project, making people available to do the tasks and any other strings that need to be pulled in order to make things happen.

I'll have more to say on this subject later as this is about all I can muster in the limited time I had for this "lunch & learn" session.

Managing the application threads

2009-11-05T07:52:00.000-08:00

Threads and multi-threaded programming is not a trivial subject. But some application developers find themselves having to start their own threads as part of their application.

I'll start this post off saying that using unmanaged threads is never a good thing. When using threads developers should be looking at how to manage all their threads in their application. Never exit the application without terminating all of your threads. Why would we need to do this?

An interesting problem is when an application is stopped (not the JVM/app-server but just the application), a new version is deployed, but the restart fails because there are still threads hanging around from the previous version of the applications and older versions of the same classes. I never would have imagined this problem occurring but I can see that if the application lost track of its threads that it would not be able to close them when it shut down.

Java provides the capability to manage all the application threads by using thread groups. When creating a thread and assigning the application's thread to a thread group all the application has to do is capture the "application stop event" and use the stop method of the thread group thereby stopping all the threads in the thread group.

The beauty of this solution is that the application can reliably manage all of the threads it creates though a single interface. It is also portable which means your application will work with any application server.

High availability through parallel cells

2009-11-05T06:06:00.000-08:00

High availability sites have unique requirements that are not easy to solve. However, one solution does revolve around using multiple cells in production. This allows for easily taking a cell out of rotation to make changes. Then when the cell goes live and if the changes didn't work it is just as easy to take the cell out of rotation again. I've used this strategy at many, many 24 x 7 sites.

Here is the link to my latest article on multiple cells.

The only problem that arises is if things like database schemas change in the backend which also means application code changes. Those kinds of changes require more work and possibly parallel schemas until the transition is complete.

Running out of disk space

2009-10-19T13:50:00.000-07:00

An event occurred this morning that reminds me that performance problems are sometimes environmentally driven. One application went down with various errors related to the file system. After some investigation it was discovered that the /tmp space ran out of space. Yet no one knew it had happened. Turns out that no monitors are in place to watch for resource exhaustion like low disk space.

One could argue why did /tmp fill out if daily (automated) housekeeping tasks were in place (they were not). But that is a pointless argument if someone moved a whole bunch of files in /tmp. They could have moved files in there to install them and then delete when the install is done. Or they could have been log files being transferred to another environment via ftp. In any case, an automated monitoring tool should have alerted the operators that disk space has come close to exhaustion and need to take action to remedy the situation.

If automated monitoring is not in place then the production applications that make use of /tmp space will fail causing an outage. And without alerts about low disk space it can take several hours for the operations and troubleshooting teams to figure out that the application failed because it could not write out a temporary file. A several hour outage that could have been completely avoided had the right resource monitors been in place.

Other resource monitors should be looking at CPU utilization, page file activity, etc.

IPv6

2009-09-30T08:36:00.000-07:00

Last year I blogged about handling ipv6 lookups that caused threads to wait thus slowing them down. Not surprisingly I've come across this a few times since then. What is interesting is the effect it can have on throughput and response time. Testing showed that once -Djava.net.preferIPv4Stack=true was enabled the test had 8% improvements in both throughput and response time. This is truly interesting data. That is a big difference and a good reason to periodically take thread dumps even if everything seems to be running nominally. One never knows when they may uncover a gem of a performance boost like this one.

PMI = all is not a good setting for production

2009-09-24T09:34:00.000-07:00

I am reminded occasionally when debugging production issues that setting the PMI level in WebSphere Application Server to "all" is not a good thing to do. At the "all" setting one can see an application exhibit negative behavior. I am also learning no two applications will always exhibit the same problem. Some applications crash and burn under the weight of PMI=all incapable of providing a response to any request. Other applications seem to continue to function nominally but the CPU for those processes seem to be considerably higher.

Finally, regardless of the PMI settings configured in the production environment it is imperative that all members of a cluster are set to the same values. Having some processes set to one level and other processes set to another results in higher CPU utilization from some JVMs and not others.

1 minute garbage collection cycles

2009-09-23T20:34:00.001-07:00

Does your application use RMI? Are you seeing garbage collection cycles with exclusiveaccess every minute (60,000 ms)? You may need to apply the following two parameters to the JVM command line.

-Dsun.rmi.dgc.client.gcInterval=360000000 -Dsun.rmi.dgc.server.gcInterval=360000000

Essential log attributes to log - duration of a request in access.log

2009-09-22T17:13:00.000-07:00

It is absolutely imperative that the access.log file from your Web server record the total response time as measured by the Web server. This is available in all flavours of Web servers. It provides definitive response time numbers as they leave the Web server that can be used as a second data point to the application monitor's servlet response time. The log entry also allows sysadmins a chance to put their log monitors on it to alert on long response times.

Default values - they're not for everyone

2009-09-22T16:56:00.000-07:00

I am frequently asked about about the significance, or lack thereof, of default values to one or more configuration items. People need to remember that default values are simply starting points so the environment can be brought up. For example, an operating system has the default value for TCP Keep-Alives set to 2 hours. According to RFC 1122 this is an acceptable default value. However, if you look at the acceptable ranges of values it starts at 10 seconds and can be set as high as 10 days. So, obviously, the default value is not going to work for everyone. Some sites may need to set it low to around 10-15 seconds. Other sites might need a 2 or 3 day setting.

Additionally a comment was made in general about setting the value of TCP Keep-Alives can not be lower than 2 hours. I think it was a misreading of the RFC specification that reads "This interval MUST be configurable and MUST default to no less than two hours." Read that the DEFAULT must not be less than two hours. This does not imply that the value can not be lower than two hours. The lesson learned there is to read the specifications to the letter. Unfortunately I think the emphasis in the RFC on the word "MUST" does distract the reader from the word that follows "default" and can subtly mislead the reader. Erroneous information is consequently passed on as a rule. Unfortunately, if no one backtracks reads the RFC and verifies the rule then rampant disinformation is spread and becomes written in stone.

Trust but verify every setting in your environment. Every setting should be tested as thoroughly as possible. Documentation should then record what was tested and what values were selected over others and why. This documentation will be valuable to the group of people maintaining the application 5 years from now and I can guarantee it will not be you the reader. It will be whoever is watching the store after you have moved on to new and exciting career opportunities.

Enable verbose GC

2009-09-16T08:31:00.001-07:00

http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21114927

I really recommend running in production with verbose GC enabled. The use of the term verbose is unfortunate as it is not as verbose as people think it is. And the data collected is invaluable.

If you have any doubts, enable verbose GC in test and see if you can measure any impact from enabling it.

socketread0 timeout on JDBC connections

2009-09-16T08:11:00.000-07:00

Since I run into this problem every once in a while I'm putting up a note here to help others when they run into this problem. JDBC calls can sometimes get stuck on socket read calls to the database if some rather nasty network problems exist. The only way to determine if network problems exist is to use tcpdump (AIX, Linux) or snoop (Solaris) to capture the packets. One can then use Wireshark (or its predecessor Ethereal) to read the capture files. If you see issues like "unreassembled packets", "lost segments", "duplicate ACK" or checksum errors then most likely the network is having some abhorrent behaviour affecting the server. If random threads hang on socketRead0 calls that never seem to get a response then the only way to deal with this is through timeouts.

On DB2 follow use this parameter:

http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=/com.ibm.db2.luw.apdv.java.doc/doc/r0052038.html

blockingReadConnectionTimeout
The amount of time in seconds before a connection socket read times out. This property applies only to IBM Data Server Driver for JDBC and SQLJ type 4 connectivity, and affects all requests that are sent to the data source after a connection is successfully established. The default is 0. A value of 0 means that there is no timeout.

For Oracle use:

oracle.jdbc.ReadTimeout

http://forums.oracle.com/forums/thread.jspa?messageID=2326985

Logs, log identification, log monitoring

2009-08-18T21:15:00.001-07:00

Logs are an ultimate source of information to the health of an application. Logging data helps understand if something has gone wrong and what might have gone wrong. The problem is that there are two sides to logging.

Developers provide the first level which are the log statements themselves. I have always found that a major/minor log code (i.e. (error/warn/info/trace = major; minor = module/method/logic) to be the most straight forward way to provide log information. Identify the type of log message being recorded to a specific category.

For the operations side of the house they can monitor logs and based off of known major log codes they can decide if they need to take action. A major code that indicates an error can be immediately dispatched to a field technician. While other levels can be monitored or collected if in troubleshooting mode.

However, if an application development team does not provide any well defined codes for log messages and only provide free form text then the operations/runtime side has difficulty determining if a specific log entry is a problem or not. Unfortunately, most application development teams choose the free form text logging as their solution which makes runtime operations that much more difficult and, in many cases, confusing.

Keeping notes (AKA your operational run book)

2009-07-28T06:37:00.000-07:00

This morning I was reminded about the importance of keeping notes about the problems I encounter and the solution that provided the fix. A co-worker of mine researching a problem came across my blog post through a google search. It seems their environment is suffering from the same symptoms. Had I not kept a note about that issue they would have most likely gone through the same level of effort I originally did to research and find the solution. Hopefully when they apply the fix it resolves their problem (I think it will) and they didn't have to spend several hours/days to get to resolution.

This brought to mind the importance of a run book. If you are not familiar with the term a run book is basically an operations manual. It spells out what needs to be done for various tasks. For example, if we need to deploy a new application into production the run book has a full set of instructions (a recipe if you will) on what to do. Likewise, when the trouble shooter debugs a problem they record in the run book what the problem was, the steps they took to determine root cause, the fixes they tried and which one(s) finally worked. This way should the problem reoccur and a different shift of people see the same problem they can refer to the run book and go through the same steps.

The run book does not have to only address technical details. It can also provide operational response instructions like how to run a war room,who needs to be involved, when various organizations/teams are engaged, what triggers an engagement, etc, etc. Every operational detail can be recorded in the run book.

Does your production environment have a run book?

Finding information (like tuning guides)

2009-07-20T08:33:00.000-07:00

I frequently get queries about finding information on one topic or another (generally around performance or application monitoring). A co-worker in Australia was looking for some general WebSphere Web services tuning guides. IBM.com has a wealth of information on a variety of topics so I thought I would put together this post on how to search for information.

This particular google search is one I used to find information for Web services tuning.

You'll note the key in the search string is the beginning "site:ibm.com" which restricts the search to only those items found at ibm.com.

Latest article

2009-07-20T08:31:00.001-07:00

I forgot to post when my dW article came out. This is a new series I'm going to write on defensive architectures. Part 2 is being reviewed by my colleagues right now.

Problem Determination - High CPU strategies on Windows OS

2009-07-09T12:37:00.000-07:00

I've been quiet the past couple of months but that is because I have an article coming out on deveoperWorks soon. I'll post a link to that when it is ready.

A colleague called me today to talk about a high CPU scenario and steps to take to try and resolve it. My first recommendation was to get thread dumps of the JVM when it hits the high CPU scenario. Then I found out the JVM is running on a Windows OS. Since it is not possible to execute a kill -3 on Windows and one has to use the script methodology with no CPU available it is tough to get a thread dump.

I suggested he collect thread dumps as the CPU starts to climb. This works if they can (a) predict when the CPU will start to climb and (b) it climbs slowly enough to be able to collect javacores along the trajectory. It sounds like the CPU can spike in a matter of seconds even at stead load volumes. Ugh.

My final thought is to reduce the thread pool max in half and rerun the test. Perhaps there are simply too many threads executing and nothing will prevent it from going 100% CPU. Though I have seen code that can drive even an 8-way to 100% with just a few threads. I'm waiting to hear back. But I think the latter strategy of reducing the thread pool max will be the best course of action for them. I expect an email in the morning.

Predicting Capacity

2009-03-26T15:13:00.001-07:00

I have had a number of calls around "We have an application, not built yet, but we want to know if we have enough capacity and/or will the application scale/perform in our environment."

My analogy to this is the following. Someone shows me the blueprint to a Formula 1 (or NASCAR) race car. They ask me "will this car win the race?"

While the car being proposed may have all the right components (wheels, engine, gearbox, etc) there are other unknown variables that determine if the car can win a race. Without extensive testing there's no way to know if all the components are assembled correctly nor if they are configured to work together in optimal manner (performance tuned).

Related to this is the ability of the car to both get off the starting line and finish the race (a prerequisite for winning). Again, there are a number of variables that impact this. First is some catastrophic failure in one of the components. Then there is the skill of the car's driver. Then the other drivers on the same racetrack and crashing into our car.

There is simply no way to predict if the car can win much less finish a race.

The same attitude has to be taken when planning your production IT environment. You can't predict behaviour. You can, on the other hand, conduct stringent system integration and performance testing in order to see if your application will (or will not) perform as expected in production.

This is why everyone should test as early in the development cycle as possible. I was working with one application that had been in production for several months with constant, recurring failures. There were fundamental application design problems in how the application was developed. It took us four more months to fix those design problems and get the final, new version of the application into production. Don't put off testing. Do it right. Do it early. Test early. Test often. That is the key to success in production.

Too many testers?

2009-02-18T03:57:00.001-08:00

Wow. I never thought I'd see a statement so outrageous. Someone, in this case Microsoft, has too many testers? Can this be true? Is this why Windows has fix after fix every week because their testing is so good because they have enough testers that they can get rid of excess testers?

I can't believe that is even remotely true. Most organizations suffer from having not enough testers. This is why applications deployed in production suffer. Something had to be cut out of the test plan because they either couldn't test something or didn't have the time/people.

A technology enterprise should not cut technical folks out of their organization.

I, Cringely - Cringely on technology

"There are too many testers"

When Performance Counts

2009-01-29T10:03:00.000-08:00

Macys.com holiday sales soar 26%

A site that performs well helps your bottom line. Note in the article during the critical December 2008 month sales increased 39.1% over December 2007. Obviously several factors play into this kind of success. The site had to remain available so users could access it. The site had to perform in order for users to stay connected and not go somewhere else.

Golden Rule #1 - It is what you don't test that breaks in production

2009-01-20T04:45:00.001-08:00

I'm putting together information for some folks around performance. So I'm coming up with a set of golden rules. My first one is:

It is what you don't test that breaks in production.

This isn't specific to WebSphere products either. This goes pretty much across the board.

I work with quite a few people on performance issues. There are two types of problems that could have easily been avoided had the testing been conducted properly.

(a) 80/20 rule

Some people live by an 80/20 rule. We'll test the 20% of the application that 80% of the users use. Um, what about the other 80% that goes untested? What if that brings down the site even if only a single user hits it? I'd rather keep the site up and test everything. Wouldn't you?

(b) Boundary value problems

Everything takes input. Not every application validates input. This leads to problems with applications running unbounded database queries because the filter wasn't filled out correctly. Or someone fills in all 250 rows on a page and crashes the site because the testers missed that point. Every use case has boundaries. Test the zero case, the in between case and then infinity. To infinity and beyond my friends! That is where we can take our sites if we do the testing right!

Winter Hibernation

2009-01-14T07:55:00.000-08:00

I know. I've been lax on updating this site. Partly because of the online holiday shopping season (I'm always busy that time of the year). Partly because of winter (we got socked in with 5" of snow this morning).

But I will come out of my shell soon and start commenting on new things to cover for 2009. We have some interesting Process Server testing strategies to talk about too which I'm absolutely excited to share with you.

Today my focus is on getting my speaker proposals in for the IMPACT 2009 conference.

Less is more - shared resources and MAX sizings

2008-11-08T07:58:00.000-08:00

Less is more. I don't know who to originally credit for that saying and when it comes to shared resource (i.e. thread and JDBC connection) pools this is the motto to live by. Need more than 40 threads to drive 100% CPU then the performance expert needs to look at where the bottleneck is in the application. Think the application needs more than 10 connections to a database? Think again. Too often I come across situations where the application is pegging the CPU and/or the database is so overwhelmed in production we can't bring it down to failover. Next time the team wants to change parameters to shared resource pools encourage them to make them smaller. They'll look at you as if you're crazy and think you'll be the next one to lose your job ... but you won't when they find your solution was actually the correct way to go.

Speaking of which, Alex has roadmaps he didn't work on this past week (well, I did have eye surgery and a hard disk crash both within days of each other).

Why Alex needs a backup

2008-11-06T07:53:00.000-08:00

Because Alex's hard disk failed. Alex actually does have a back up of his data but not all the applications so it is going to be quite a bit of effort to get a C drive with what I need on it. That said, I'll start with a fresh Windows XP image and may have lots better performance now!

Disaster Recovery - why you need a plan and execute on it

2008-11-04T11:11:00.001-08:00

Problems happen. Fiber cuts can occur out in the wild. Hardware fails. Routers go down.

The last thing you want is to have no backup. Put the backup in a separate data center facility on separate ISP inputs. Have multiple electrical feeds into each data center.

Otherwise you will have a site outage and there will be nothing that can be done to fix it because you are waiting on someone else who you have no control over to fix the problem.

Save those (test) results and logs

2008-10-28T09:08:00.001-07:00

Organizations typically have well defined archive policies for the production logs and data. However, a lot of test environments are ignored. There is nothing more frustrating than trying to look at test results from a couple of weeks ago and the logs are gone. Thus any sort of analysis (i.e. pulling the verbose GC data from native_stderr) can not be conducted. That means the test was inconclusive.

If your organization does not have archival procedures and processes for the test environments (at least the performance one) make sure that it gets on someone's agenda.

Coding for resiliency : long running (or infinite) loops : stop the madness

2008-10-24T08:16:00.001-07:00

....
while(exception instance of MyException) {
exception = (MyException)exception;
}
....

This was code (not verbatim) that was found in a production application. Please, always have a valid exit condition for your loops. And then a 2nd way to exit the loop somehow if the primary condition always reverts to true.

Coding for Resiliency : thread locals and cleaning them up in a finally block

2008-10-24T04:35:00.001-07:00

I haven't updated this site because I had a brief vacation that was cut short by a blocked tear duct. I'm still suffering with this and the medication seems to have alleviated but not rectified the problem. Unfortunately that means going to an eye doctor on Monday if it hasn't cleared.

A colleague of mine is working on a problem where they plan to use a thread local within a servlet filter. When I read this it reminded me of another problem; uncleaned thread local variables. The key here is to be sure to clear out the thread local in a finally block before exiting the servlet filter. This is because it is very likely that a thread in WebSphere Application Server will be reused by subsequent requests (especially in high volume environments). If the thread local is not properly cleaned up during an exception situation then the next request on the same thread will be looking at a thread local in an invalid state. So use those finally blocks for what they were purposed for!

HTTP Session failover and performance

2008-10-09T07:02:00.001-07:00

One of my colleagues wrote a good article that covers HTTP session failover. There are performance implications to using HTTP sessions and persisting them. This spurs me to start a series of articles to discuss some of the pros/cons with these mechanisms.

Before session persistence existed there was no fail over capability for sessions. This is, though, probably the most performant deployment. Sure, users have to log back in but the application server doesn't have to spend time on serializing objects and then pushing them out to a database. You save on both memory (for the serialized objects) which then have to be garbage collected which is paid for through CPU utilization.

If your business requirements can stand it, and require users to relogin in a failure, then this is a good deployment strategy. I know a number of intra departmental applications that can tolerate such activity.

The WebSphere Contrarian: Back to basics: Session failover

My preferred alternative is to rely not on session distribution, but instead to rely simply on HTTP server plug-in affinity to “pin” a user to an application server, although this does mean that stopping an application server JVM will result in the loss of the HttpSession object. The benefit of doing so is that there is no need to distribute the session objects to provide for HttpSession object failover when an application server fails or is stopped. The obvious down side is that a user will lose any application state and will need to log back in and recreate it, and this may or may not be acceptable for your application or business requirements. I'll mention that I’ve worked with a number of customers that in fact agree with this view and make this their standard practice.

pClusters on JDK 1.4.2

2008-10-08T11:55:00.000-07:00

When enabling pClusters on the 1.4.2 JDK be sure to enable gcpolicy:subpool otherwise it won't work as expected.

Redundancy and unreliable software

2008-09-25T12:47:00.000-07:00

Redundancy is often a great thing to have and has save many a production environments. However, redundancy works if the underlying hardware and software work. Making an unreliable environment redundant simply means that the redundant system is unreliable too. Failing over from one failing environment to another that will fail is not going to help availability.

Therefore, the best thing to do is to make sure that the system being copied is reliable in and of itself.

Deploying with Resiliency - shared libraries testing

2008-09-18T12:49:00.000-07:00

I've been talking about Coding for Resiliency but figured I should do a little focusing on Deploying with Resiliency today.

Anytime one pushes out a new release candidate it should be tested in the same configuration as it will be in the production environment. This means that any shared libraries must be deployed out with the applications that are using them when multiple applications share the same app server.

The best that can happen is that the production environment will remain stable as long as what was tested is what is in production. However, the worse that can happen (and it will) is the site will become unstable. That is not the end of it though. Whatever deployment was put in place has to be backed out (here is where multiple cells really helps you out and if you don't know what I'm talking about go to this great article by my colleague Peter) and the previous deployment has to be re-established in production. The decision to revert also has to be made. This is, depending on the people running production, a difficult decision to make. Even more difficult if a back out cell is not available.

Therefore, depending on when it is discovered that the production site is unstable and the decision to back out is finalized it can be several hours before production can be stabilized.

This is one reason why it is important to test prior to deployment in production and that "testing in production" is never a good idea.

Code for resiliency - validate input

2008-09-12T11:05:00.001-07:00

Scenario 1
I was speaking with one of my colleagues about problems they were seeing with an application. It seems that if the user typed alphabetic characters where a number was expected the application choked, died and rolled over.

Solution
I was amazed. "Don't they validate their input?" I asked. Depending on your view point surprisingly they didn't.

Always, always validate user input. If expecting a name make sure it doesn't contain characters you don't normally see in names (i.e. Joe123). If expecting a zipcode make sure it is formatted properly. But never just take the input from the user and pass it on to the next layer.

In addition, validating input will help prevent XSS attacks.

Scenario 2
You're running a B2x Web service and have complex XML documents. But modern day parsers are vulnerable to malformed XML that can contain circular references.

Solution
Acquire DataPower and have it front your Web services. DataPower has incredibly good XML validation and is able to prevent DoS attacks from malformed XML.

This one keeps on going and going

2008-09-11T06:31:00.001-07:00

I blogged about this problem a year and a half ago. It continues to live on. If you are writing non-batch applications then set this setting to unshareable. Yes, it is unfortunate the default is shareable but we need to deal with it. This descriptor seems to move around so look for it either in the JDBC or thread section of the descriptors.

Alexandre Polozoff on WebSphere Performance: Day One SHAREABLE JDBC connection setting

change the descriptor to UNSHAREABLE

Code for resiliency - circuit breakers in loops

2008-09-11T05:57:00.001-07:00

Scenario
One common problem I run into on different critsits are run away applications. You know the type. The application is sitting there in a tight loop allocating 1 or 2Megs of data on each iteration chewing up CPU and causing the garbage collector fits. Sometimes the code is buggy. Other times the data has changed and has pointers to larger data sets than it should. A few times a new use case was discovered.

Runaway loops in an online, web application are a bad thing. If enough users go down the same use case the application can have several threads all running away. The CPU will be pegged at or close to 100% and the verbose GC data shows tons of memory being allocated at very fast rates because the garbage collector is busy collecting it!

As more users go down the same use case more and more threads get hung up on different application servers in the farm. Pretty soon, you're recycling application servers as it is the only administrative solution for this problem. The problem with recycling is you are also dumping users who have good transactions in progress. Because the code has run away there are no other options. Hopefully those good transactions will try their transaction again when it fails and not go shopping somewhere else.

Solution
Pick some arbitrary count limit as a circuit breaker. Once the count limit is reached the loop exits. In addition, when the circuit breaker is activated the application log that this occurred and dump some of the relevant data around that point. This way when a problem occurs someone can be alerted that the circuit breaker was activated and examine the problem.

I would implement the loop exit by throwing an exception (this is an exceptional case right?) from an if statement at the beginning of the loop. Let the exception handler dump the data to the log. Make sure something is monitoring the log for the exception and that someone is alerted to examine the problem.

Resiliency in code is not difficult. But you do have to anticipate the worst will happen. Every loop has to have a way to abort if it has executed an unreasonable amount of times. Otherwise production can be very unstable.

Unreasonable Arguments Opposed
While I tend to think I'm fairly even handed a number of people have told me this is a stupid idea. Well, let me tell you why it isn't. If the circuit breaker is hit then you know you are already in an invalid state. Regardless of the reason continuing to process the request is pointless because something is not right. If we only have 800 items in a product catalog then looping a 1000 times means something is wrong. Therefore there has to be a way to abort the request before we degrade production. As for that user's request there is no way they will be able to complete it. However, since the circuit breaker was activated and data was logged and someone was alerted not only do we know that something happened but someone is not actively working the problem to figure out why the breaker was activated.

What about transaction integrity people ask? Well, if you handle exceptions properly (and that'll be another CfR article) then you won't have any problems with transaction boundaries and everything will be rolled back appropriately.

What about open locks? Again, if you have coded your exception handling properly then all locks should be released.

What about open JDBC connections? Again, if you have coded your exception handling properly then all JDBC connections should be released in the finally block, no?

5 9s is not easy, it can be done but you have to know what you're doing

2008-09-08T17:21:00.001-07:00

Continuing the coverage of production outages that make the news it seems that the London Stock Exchange had a serious outage today. Most likely due to the high volume resulting from the US Government takeover of Freddie & Fannie.

We at IBM often put together high volume environments that have high availability requirements. In order to do this, and do it well, one has to make sure they have built in resiliency, enough capacity and then disaster recovery for business continuity. I've worked with a number of household name companies, world wide, on providing just such capabilities. It is disastrous when a e-commerce retailer is unable to sell product during the holiday shopping season. Things can get particularly bad for financial institutions when money is on the line.

I can't say what they did or didn't do but it certainly seems like people want answers. Reassurances are going to be hard to come by until they do a lot more ground work.

London Stock Exchange crippled by system outage | Reuters

The exchange would not say whether volume was the issue and declined to give details on what had caused the problem. But angry customers were demanding an explanation.

"We want answers as to how this happened in the first place and reassurances that it will not happened again," said Angus Rigby, chief executive of brokerage TD Waterhouse.

debug logging has no place in production

2008-09-05T10:52:00.000-07:00

You know, I'm often confronted with this problem and I have yet to really understand why it exists. Lots of people are running applications in production and they have logging set to the debug level. From a performance perspective this is intolerably horrible! You're constantly hitting disk, garbage collecting spurious strings, serializing around the disk access that it just makes no sense to me how I keep getting javacores with stack traces in logger.debug (or worse, SystemOut.println!)... speaking of println, one should never, ever use println for logging. There is no way to control it like a logger. With JDK v1.4 (I believe, maybe it is Java 5) there is a logger in the JDK. Use it! And set your log level to WARN or ERROR but nothing more granular than that unless you're debugging a problem.

tprof is your friend

2008-09-04T14:48:00.001-07:00

I've collected data on various 100% CPU problems and always am amazed how useful tprof data is yet it is very unintrusive.

IBM - MustGather: 100% CPU usage on AIX

Collecting data for 100% CPU usage problems with IBM® WebSphere® Application Server on the AIX® operating system. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.

Disaster Recovery is not easy (just ask the FAA)

2008-08-27T06:45:00.001-07:00

I commented on the flight plan problem yesterday and it is interesting to see that the FAA actually had a Disaster Recovery (DR) plan and running DR site in place. But it didn't work.

U.S. Airports Back to Normal After Computer Glitch - NYTimes.com

The other flight-plan facility in Salt Lake City had to handle the entire country when the Atlanta system failed but the backup system quickly overloaded, Brown said.

The interesting thing to point out here is that while the FAA had a DR plan in place it seems no one looked at the capacity of the backup site. Was this bad planning? Bad execution? I wonder, did they ever test their DR environment? It seems kind of pointless to have spent the time and effort to build out the DR environment to then have it fail when it was needed. A waste of tax payers dollars IMHO.

Unfortunately a DR site failing to accomplish what it was intended to do is more often the case than not. First off, building a DR environment is not easy. I know some pretty smart people that have gotten this wrong. Secondly, after the DR environment is built it must be tested and tested as if a real disaster has occurred. The best corollary I can think of is testing your backups. Does anyone ever restore a server to see if the backups are good? I once was working with an organization that had their servers crash due to a hardware failure. They went to recover the servers and only then discovered the backups were corrupt. It took them a few days to rebuild the servers so I got to go home.

On a tangent, seeing that this is a blog on WebSphere Application Server performance, the one mistake I have seen people make with DR from a WebSphere Application Server perspective is to have a cell cross data center boundaries (I'm sure I've blogged about this before but here it goes again). The reason this is a mistake is that networks are not reliable. TCP/IP is not guaranteed, reliable delivery. Any hiccup which can include dropped packets, static electricity or planetary alignment in the solar system that causes even a slight degradation or lag in the networks between the two data centers can reek complete havoc within the cell. And guess how hard it is to troubleshoot that problem? Yeah, tough, real tough. Thus, even when a disaster is not occurring strange "problems" can occur with the applications running in that cell that just can not be easily explained. And the more distance you put between the data centers the more likely that strange problems will occur.

Likewise, when a disaster occurs and half the cell disappears this alone can cause other problems. For one, the application servers left running in the other data center will be looking for their lost siblings and spending time, CPU cycles, RAM and network bandwidth searching. This too affects the applications running in this configuration.

Therefore, moral of the story is to never have the cell boundaries leave the data center. In fact, there are a number of reasons one should be running multiple cells within the same data center to better manage planned and unplanned maintenance. Particularly in high volume, business critical, high availability environments.

Oh, that, and hire a real good DR expert if you're planning on implementing DR. Nothing like having the DR plan fail. In the FAA's case there are no repercussions (i.e. fines imposed that cost them millions of dollars a minute). Granted, this probably did cost the airlines and the unfortunate passengers money but nothing the FAA will have to reimburse. For a lot of private enterprises there could be severe repercussions not just in terms of penalties/fines but customer loyalty, customer trust and how your enterprise is viewed as a reliable business partner going forward.

Network performance related issues: isolate the network

2008-08-27T06:14:00.001-07:00

Over the years I've worked on a few problems revolving around the network itself. What I'd like to do is first throw down a gauntlet around performance testing environments and the #1 factor is to have an isolated network. This means that the only traffic going across the wire/routers/switches are only related to traffic in the performance test.

For example, one time we had spent the day troubleshooting functional problems in the application. We finally got a good build and set up to start our baseline performance test later that night. We kicked off the test around 20h00 and about 2 hours into the test our response times were tanking into the several seconds range. I could not find any problems on the application server or the Web server tier. A little digging around and I found out the load generators were placed in a location remote from the test environment. At around 22h00 network backups kicked off as it was their policy to backup their servers at night. Good idea to run backups at night but because our load generators were sharing the same network their response times went down the drain. We had to break off our test and we all left the office around 23h00 when it was clear that it would take several hours to relocate the load generators onto the same switch as the isolated performance test environment.

There is nothing more disappointing than gearing up for a test in the evening only to have it been a waste of time (both mine and the good folks I was working with) much less keeping everyone away from their families. The reason the load generators were remotely located was because managers did not give the local team the time they needed to move them. As it was, we had to move them and had to take the hit to the testing schedule to do it.

Moral of the story, make sure every piece of the performance test environment is isolated from the rest of the organizational network. Otherwise testing will be inconclusive.

Root cause analysis

2008-08-26T14:53:00.001-07:00

It is vitally important when problems occur that the root cause is identified. If it isn't the problem will reappear again.

Though my guess is with the cutbacks in future flights we won't see this problem for a while.

FAA computer problems cause flight delays - CNN.com

The problem appeared similar to a June 8, 2007, computer glitch that caused severe flight delays and some cancellations along the East Coast.

Do you have the capacity for holiday shopping season 2008?

2008-08-26T06:04:00.001-07:00

Ironic is probably not the right term. I believe there was a problem in the air once over Canada over a similar situation where the plane was running out of fuel because the guy tanking up the plane was using one unit of measure and the person ordering the fuel used another.

That probably isn't the case for Amtrak. But this story brought to mind a question... do you have enough fuel (capacity) for the upcoming holiday shopping season? It is the end of August. By my rough estimate we probably have about another 2 months before the 2008 online holiday shopping season begins. If you haven't already run the numbers on your capacity I would highly recommend at least reviewing them right now. Be sure you have the horsepower to let your end users have a pleasant shopping experience.

Amtrak train runs out of fuel, stranded 2 hours - USATODAY.com

It was the little engine that couldn't — because it was thirsty for fuel.

Otherwise you could find yourself having some pretty severe strain in the server environment.

Test out the different GC algorithms

2008-08-26T05:53:00.001-07:00

The latest versions of the IBM JVM provide a number of different Garbage Collection (GC) algorithms. Since no one algorithm is always the best to use it is imperative that the performance test plan allows for testing that cycles through each of the GC algorithms, adjusts the parameters based on the verboseGC output and sees if the results help improve the application's performance.

For example, I have noticed on a number of engagements around Process Server that using the gencon (generational garbage collection) has a positive effect on the server's performance. Obviously tuning the nursery and tenured spaces is necessary based on each individual application from the verbose GC output. But the fact that Process Server always seems to run better with gencon it is one of the first things I like to test out.

On the flip side, there are some tools out there that dynamically create object classes. My understanding is these are used for transformations but what I don't get is why they need to be dynamically created. Most object transformations are from one static object model to another. Once the transform is generated to continually generate another duplicate object makes no sense. Anyhow, I digress... the point I'm trying to make is that when dynamically generating lots of object classes (oh, say about 10,000 per second!) this can create serious havoc with the GC algorithms. In this particular case of 10k/sec it actually ends up looking like a native heap leak when using gencon. Thus if you're doing something like this (and I hope you're not but that is a different posting) then you may have to look at one of the GC opt algorithms instead.

Operational stability

2008-08-25T13:42:00.001-07:00

One aspect of performance problems occurs with runtime operations and the stability from a runtime perspective separate from application code or content changes. Some of the things to consider when building out a 24/7 site and the ability to maintain availability even when things take a turn for the worst. I assume you already have built out multiple cells for the deployment of your high availability environment. I prefer cells over clusters because you have more operational/runtime flexibility with cells than you do with clusters particularly when you are trying to apply maintenance (i.e. shut of the load balancing to the cell that is going to be updated).

1. Ability to make repeatable changes to the configuration. This involves scripting and testing the scripts (preferably not in production) to be sure they work as intended.

2. Identify what is the active configuration. This is important to understand which configuration is active such that the correct cells are taken out of rotation.

3. Make the scripts aware of the active configuration. One really doesn't want to have scripts making changes to the active configuration by mistake.

4. Back out. Depends on your requirements but being able to flip back to a previously working configuration as quickly as possible minimizing downtime.

fn:toLowerCase

2008-08-20T06:47:00.001-07:00

Ran into a stange situation where starting up the app server resulted in an exception. It appears the error manifests itself thinking there is no function mapped to the name fn:toLowerCase. This started happening after fixpack 19 was applied over fixpack 11. Long and short of it is the solution was to clear out the tmp files so the JSPs were recompiled after applying fixpack 19.

00000042 ServletWrappe E   SRVE0068E: Could  <br />   not invoke the service() method on servlet /global/tiles/g  <br />   eturl.jsp. Exception thrown : javax.servlet.ServletException: No  <br />   function is mapped to the name "fn:toLowerCase"  <br />           at org.apache.jasper.runtime.PageContextImpl.handlePageException  <br />   (PageContextImpl.java:650)  <br />           at com.ibm._jsp._geturl._jspService(_geturl.java:192)

Collect traces when having problems with the JDBC connection pool

2008-08-18T08:37:00.001-07:00

The referenced technote below provides the instructions on how to collect JDBC traces when encountering certain JDBC connection pool problems.

0000006b FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/abcd.

If you see the above error message in your logs we really need to collect JDBC traces. The traces will tell us why we are seeing this error. Is it long running transactions? Not a high enough max? Connections not properly closed? Who knows... the trace knows. Collect the trace and analyze the problem or open a PMR. Don't just blindly increase the connection pool maximum without knowing why they are being increased.

IBM - Using Connection information in WebSphere trace files to troubleshoot J2CA0045E and J2CA0020E or connection wait time-out problems.

J2CA0045E and J2CA0020E errors can be caused by many problems. They are showing a time-out condition where a resource or a managed connection is not available to fulfill a connection request.

In this technote we will use connection information in WebSphere® trace files to troubleshoot J2CA0045E and J2CA0020E or connection wait time-out problems.

3 days of not selling burgers

2008-08-16T07:35:00.001-07:00

Wow, a 3 day outage at Netflix. I know this is a blog about performance (specifically around WebSphere Application Server) but I have a fascination with production outages because I normally work those kinds of problems. They always revolve around human error either by fumble fingering something or not executing (like not conducting performance testing).

But a three day outage is excessive. That must have been a real interesting problem because I've never seen an outage that long (at least not after I have arrived to fix it). And it'll cost Netflix an estimated $6 million.

As the article states, imagine if McDonald's couldn't see burgers for 3 days. BTW, the McD's in Dwight, IL off I-55 lost their cash registers one day a couple of weekends ago when I stopped in to get some food on my long drive to Chicago. It was interesting to see how dependent McD's is on that register system because it drives their whole operations and what the human equivalent (i.e. yelling orders into the back area) and having to use calculators and paper/pen to record how many of each item was sold. Needless to say the credit card readers were down too so if you didn't have cash you were going hungry that day.

Lessons From Netflixs Fail Week - Bits - Technology - New York Times Blog

Netflix, the DVD-by-mail service, largely ceased shipping DVDs to its 8.4 million subscribers for three days this week. The company vaguely blames a technology glitch.

One reason I like to take thread dumps during performance testing

2008-08-13T05:52:00.001-07:00

I've written before about thread dumps and the value of taking them during a performance test. The other day we finished our baseline testing so we started a duplicate test simply for taking thread dumps. Even though I wasn't seeing any anomalies in the baseline this is just something I do to make sure I dot all my i's and cross my t's. Plus, if there are any anomalies they will show up in the thread dump.

Lo and behold in the thread dumps (remember we take at least 3 thread dumps spread at least 2 minutes apart) I found a number of threads sitting on

at java/net/Inet6AddressImpl.lookupAllHostAddr(Native Method)

which seemed odd to me. I live by the rules of mathematics and its definition of randomness. A random thread dump at any random point in time should result in the threads doing random different things in each thread dump. If one thread dump in the series shows a couple of threads doing the same thing then that is odd. If more than one thread dump in the series shows more than one thread doing the same thing then we have a bottleneck! Bottlenecks can limit an application's ability to use CPU and keep the throughput down. If you can't fix the bottleneck then you'll need more hardware to scale up which means spending more money. If you can afford that then stop here and call your finance guy.

I searched the PMR database and found that indeed there is an interesting side effect to IPv6 and it was affecting the throughput of the application I was testing! Fortunately the PMR referenced a technote on the subject and I'm hoping we can eliminate this issue. The good thing that will come out of this is we will see a throughput improvement in the application once we apply the proper configuration. The improved throughput will mean an immediate cost savings in additional hardware we would have had to purchase to make up for the differential. A win win for everyone (well, except for the IBM hardware sales folks but c'est la vie!).

Although in this particular multi-tiered environment (Process Server talking to WebSphere Application Server talking to CICS) I still have to go back and collect thread dumps on Process Server once I'm satisfied I am not seeing any other issues in the WAS tier. Who knows what anomalies that will uncover (hopefully none so this testing can wrap up soon).

IBM - HostName lookup causes a JVM hang or slow response

If the DNS is not setup to handle IPv6 queries properly, the application must wait for the IPv6 query to time out.

Test early - Test often

2008-08-13T05:31:00.001-07:00

There is nothing I can stress more (and I'm sure I've done it before but I'll do it again) than to start performance testing with the first build of any application. If one thing that 20-30 years of computer science has taught us, particularly in the past decade of the .com boom, is that performance testing can not be just the last few weeks of an application lifecycle.

Performance testing is where the application's warts and ugliness are exposed. It takes time to troubleshoot and solve each problem as they are encountered. A lot of the problems will entail code changes that then have to be re-tested functionally. I've been on some performance engagements that have lasted months due to the sheer number of application code defects that had to be corrected.

Managers have to understand that software will always have bugs and definitely will have performance issues. This is a fact of life. No single person has the brain power to take Java code (or any other language of choice) compile it and load test it in their head. So do the right thing. Take each application build (or weekly release candidate if you do daily builds) and start performance testing from the start. This way as each week progresses you'll be able to determine if the application is meeting expected performance, non-functional requirements objectives. You'll also be able to see if the application is improving or degrading each week.

Finally, with continuous performance testing there won't be any surprises in the last few weeks leading up to the "go live" date in production.

JVM tuning - too little heap

2008-08-11T17:41:00.001-07:00

I'm working with an application this week that has been given a max heap of 256M. This is causing GC to occur every 250-400ms which consumes about 35-40ms of time each GC. In addition, the tenured space is down to 7% free. Thinks we need a high maximum JVM heap?

More outages making the news

2008-08-07T16:29:00.001-07:00

I noticed yesterday that American Airlines went down. There was discussion about it at www.flyertalk.com and I then found this article about a Google outage. This just goes to show folks how hard it really is to provide reliable Internet services to the masses. While one can only speculate how either of these outages occurred it is clear that even the big guys have a hard time doing a hard thing.

Google Gmail, Google Apps Outage in the Cloud

The search company spends billions of dollars on servers that can support the services its millions of Gmail and Apps users require;

Messaging engine and its database

2008-07-31T16:47:00.001-07:00

Some optimizations are required on database tables. My good friend Tom Alcott discusses the lack of necessity to optimize the messaging engine database tables and for good reason.

The WebSphere Contrarian: Are you sure you want to reorg that messaging engine database?

The standard practice for database administration is to periodically check on the database and table organization to insure optimal performance -- but do these standard practices apply to a database used for JMS persistent message storage with IBM® WebSphere® Application Server?

New book on DataPower

2008-07-31T06:56:00.001-07:00

If you are not aware of the performance boost DataPower can provide your environment and the XML threat protections it provides you really need to get up to speed on it. Some of my colleagues have taken the time to put this book together.

Amazon.com: IBM WebSphere DataPower SOA Appliance Handbook: Bill Hines, John Rasmussen, Jaime Ryan, Simon Kapadia, Jim Brennan: Books

IBM WebSphere DataPower SOA Appliance Handbook (Hardcover)

"free -m" on the Linux command line

2008-07-23T18:56:00.000-07:00

One thing you can never do is over commit available RAM in the machine. If the application server fails to start and SystemOut.log contains a message like this:

JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate heap. 1G requested
Could not create the Java virtual machine.

Then it is highly likely that you tried to start an app server with not enough free physical RAM. Check the "free -m" command on Linux. Otherwise refer to your OS specific manuals to see how to determine how much free RAM you actually have.

Reliability and Availability

2008-07-22T09:52:00.001-07:00

Common topics in the performance space are reliability and availability. This article goes on to describe some of the problems that can occur in such an environment. This is the challenge of building reliable systems from unreliable components. From a hardware perspective this can be done if one has enough money for all the redundancy that is necessary. If one tries to do this on the cheap they will fail.

There is an interesting point in the article that Google is trying to solve this problem with software. While there are products like WebSphere XD that provide software level solutions for some problems they can't solve the problem as easily as hardware can. For example, the database is slow. Giving the database faster disk, more RAM or a RAM backed SAN and you can eliminate that problem. There is relatively little you can do from a software perspective to fix that. Another example is a server goes down. Sure, we could route data to another server using a software component but then why not just do it from the hardware level? Okay, so there are a couple of places where software is useful like maintaining J2EE affinity but the same can be done by a hardware load balancer. It just depends on where you put the smarts.

The problem with software to try and fix this is that it introduces another, more complex, layer of hardware/software where as redundant hardware makes things a lot simpler.

A few people will argue that software is cheaper. I don't agree with that argument. I think hardware is cheaper. Especially when it makes troubleshooting that much easier (and quicker) than home built software. If it takes 6-12 months to debug software in production then that is a lot of money (and bad press earned) down the drain.

Amazon S3: For now at least, sometimes you have to reboot the cloud | News - Business Tech - CNET News

Afterward, Om Malik called cloud computing frail: "The S3 outage points to a bigger (and a larger) issue: the cloud has many points of failure--routers crashing, cable getting accidentally cut, load balancers getting misconfigured, or simply bad code. And he's right, to a degree, but there are three things that shouldn't be overlooked before writing cloud computing off as a failure.

Negative testing

2008-07-18T09:45:00.001-07:00

The following article is a good example why companies writing software need to hire subject matter experts when it comes to testing their applications. Particularly in what is commonly referred to in performance speak as "negative testing." This is where we, subject matter experts on performance testing, purposely cause a negative event to occur. For example, I routinely disable the Network Interface Card (NIC) [also known as your ethernet card] while running load/stress tests just to see how the application environment handles the event. If the application breaks then it fails the test and a defect is written up against the application and back to development it goes. It is easy enough in Unix environments to disable a NIC card but if worse comes to worse I'll pull the ethernet cable out of the jack. Crude but it works just as well.

Irish Examiner | Airport radar meltdown due to 'faulty' component

The malfunctioning network card, a component that allows computers to communicate with each other, was also blamed for previous glitches in the Dublin system.

It is unfortunate that the people that put that airport radar system didn't conduct negative testing because a problem like the one that occurred could have been completely avoided.

Likewise, while they are adding more monitoring I'm dubious that will help them. The fact that they haven't tested for negative events what other negative events they haven't thought of could occur? For example, some of the others I routinely test for are lost packets in the network, total network failure, network lag, 100%+ CPU, low memory, too many airplanes in the radar, duplicate radar images, etc, etc, etc and the list goes on and on.

All they need is for a different negative event to occur and they could (and probably will) suffer another outage. What they need to do is get a subject matter expert to teach them how to test their code.

BTW, notice the sentence about "delays were still being experienced at peak times"? Seems someone hasn't done stress testing either...

IBM Support Assistant

2008-07-16T10:17:00.001-07:00

There is an update to the IBM Support Assistant. If you are running WebSphere Application Server and you do not have this tool then click on the next link and download it.

IBM Software Support - Overview

The IBM Support Assistant is a complimentary software serviceability workbench that helps you resolve questions and issues with IBM software.

createOrWaitForConnection and why we need a finally block

2008-07-11T07:47:00.001-07:00

In previous versions of WebSphere Application Server this method actually had a name I liked better which was: createOrWaitForVictimConnection where the victim connection was one that was not closed within the same thread that opened it and eventually reaped by the app server. But either way, if you see this message timeout in your log file then that means somewhere, somehow someone has not closed a connection to a pooled resource properly. Get the following 3 words into someone's vocabulary... try, catch, finally. I can't emphasize enough how important it is to close the connection in the finally block. If you don't, then any exception that occurs can leave un-closed connections hanging around. If you are in a high volume environment you'll find this to be a serious bottleneck! Follow the following psuedo code...


Connection con;
try {
  con = ds.getConnection();
   // do some work
  methodThatUsesConnection(con);
} catch (Exception e) {
  //maybe log an error here if you like?
} finally {
  con.close();
}

Off topic alphaworks listing

2008-07-11T06:56:00.001-07:00

I try not to go off topic as this is a performance blog but some folks might find the following technology preview useful.

alphaWorks Services | IBM Pass It Along | Overview

A peer-to-peer knowledge exchange network that builds communities of experts and learners around "nuggets" of knowledge.

Do you run WebSphere? Then you need this diagnostic tool!

2008-07-02T07:20:00.001-07:00

IBM: IBM Support Assistant

I have used this tool (and its predecessors) so frequently I don't go anywhere without it/them. Not only does it produce handy little graphs like the one above showing Java GC but it also provides some darned good analysis on recommended changes to the JVM command line parameters (especially if you're running on WAS v6.0.x or earlier which do not run on Java 1.5) to improve your memory utilization. Now, of course, you can only get this kind of feedback from the tool if you followed one of my earlier recommendations to turn on verbose GC. You have turned on verbose GC by now, right? If you haven't then you have to go and do that right now.

So, go to the link for the IBM Support Assistant and download this tool.

ITCAM instrumenting your own method capture

2008-07-02T07:13:00.001-07:00

One of the nice things that application monitoring tools provide is the ability to specifically measure information about your own method calls. This page describes how to do so with the IBM ITCAM tooling.

Help -

A custom request is an application class and method that you designate as an edge or nested request. When the method runs, a start and end request trace record is written to the Level 1 or Level 2 tracing.

WebSphere Process Server Performance

2008-07-02T07:10:00.001-07:00

The first week of June I got to work in person with a colleague of mine, Richard Metzger, from the IBM labs in Böblingen on a process server engagement. Richard has started his own performance blog for process server!

WebSphere Process Server Performance

Thoughts and opinions around performance of and capacity planning for IBM WebSphere Process Server and other products (like e.g. DB2), as they are used in the context of business process management and business process automation solutions.

kill -3 does not produce a javacore

2008-06-23T07:35:00.001-07:00

This is a sporadic problem I run into when kill -3 does not generate a javacore and the process has to be terminated. The first thing to check is the service release of the JVM and ensuring the JVM is at the correct level with the corresponding level of WebSphere Application Server. In some cases I have installed later fixes than those that have been tested with WebSphere Application Server.

Another suggestion a colleague of mine had was to generate an AIX core of the process (make sure the ulimits for file and core are properly set to unlimited but you already knew that because you followed the installation instructions for WebSphere Application Server). I don't remember which kill signal needs to be sent but I'm sure a Goggle search will reveal that answer.

Get thread dumps during supplemental load tests

2008-06-17T14:50:00.001-07:00

I recently found a bug in an application that the developers were not aware of. The code had a synchronized block they thought would be low cost. Low and behold in our load testing we found that after a certain number of users were active their response times started to go up exponentially. I took some thread dumps and found the synchronized block of code.

For anyone interested in performance:

1. Load test
2. Get thread dumps during bad response times.

If neither is done there will be problems in production. If you do not know how to analyze javacores open a PMR with IBM and IBM Support can help identify the problem.

WebSphere Process Server - database configuration

2008-06-11T02:19:00.001-07:00

It is crucial that a WPS gold topology have the databases properly configured. If they are not (i.e. all pointing to the same database instance) there will be contention issues that will not resolve themselves. One of my esteemed colleagues wrote this great article.

Building clustered topologies in WebSphere Process Server V6.1

This leads you to the database settings screen, arguably the most complex of all the steps.

Why cross cell data centers are not a best practice for disaster recovery

2008-06-04T12:15:00.001-07:00

This topic has been coming up time and time again. It is time that people read about the trade offs when trying to conduct DR with a single cell across multiple data centers. Yes, this might work. But more often than not the various interconnects between the two data centers and the interaction can lead to very negative consequences that disables the intended DR effort.

Do the right thing. Isolate the two data centers with separate cells. You'll find this works not only much better but has a very high success rate if done correctly (i.e. you use scripting to build your environments therefore having repeatable processes across DCs).

Comment lines: Tom Alcott: Everything you always wanted to know about WebSphere Application Server but were afraid to ask -- Part 3

While the notion of a single cell across data centers is bad from a risk aversion perspective, running a cluster across two data centers not only requires you to forget about minimizing risk, as noted above (since a cluster cannot span cells), but further increases risk along multiple dimensions.

Selling Application Monitoring

2008-05-27T11:39:00.001-07:00

Take the word "security" and replace it with "application monitoring" in this article and you have my problem. Application Monitoring is really important.

Schneier on Security: How to Sell Security

How to Sell Security

It's a truism in sales that it's easier to sell someone something he wants than something he wants to avoid. People are reluctant to buy insurance, or home security devices, or computer security anything. It's not they don't ever buy these things, but it's an uphill struggle.

Recent performance related articles

2008-05-22T07:28:00.001-07:00

Every once in a while I take pen to paper (well, more correctly fingers to keyboard) and key out another article. Here are a couple I've written this year.

Comment lines: Alexandre Polozoff: Cultivating a performance specialist
Comment lines: Alexandre Polozoff: How well does traditional performance testing apply to SOA solutions?

Disaster Recovery & AIX GLVM

2008-05-22T06:18:00.001-07:00

IBM eServer - Using the Geographic LVM in AIX 5L

"GLVM can help protect your business from a disaster by mirroring your mission-critical data to a remote disaster recovery site. If a disaster, such as a fire or flood, were to destroy the data at your production site, you would already have an up-to-date copy of the data at your disaster recovery site."

Absolutely critical technology if you are at all interested in DR (Disaster Recovery). I know at least two customers who have used this with synchronous updates (i.e. the local disk update is synchronized with the remote disk update) and seeing little overhead even over a distance of about 150 miles between data centers. This is an ideal technology for people looking to setup DR for their WebSphere Application Server, Portal, Process Server, MQ, DB2 and the list goes on and on. I'm absolutely excited about this technology and the potential impact this can have for our high availability site customers.

If you haven't read up on GLVM I highly recommend taking the time.

JVM memory and high CPU

2008-05-22T06:07:00.001-07:00

In the "Butterfly Effect" a butterfly flapping it's wings on one side of the world can create a typhoon on the other.

In the world of Java: memory usage can cause high CPU. Summer of 2007 I spent 3 weeks working with a customer in the UK and we spent a few days measuring the memory usage of the application. We worked with the developers at reducing the memory footprint. In many cases these were simple code changes not requiring architectural or design changes to the app.

Reducing an application's memory footprint reduces the amount of garbage collection the JVM needs to execute. GC will use up CPU so obviously executing fewer GC cycles reduces the CPU load.

The UK application went from about 80+% CPU down to 25-30% for the exact same load test and better response times.

Verbose GC is your friend here too. Another application I'm looking at right now is suffering from high CPU and we can see in verbose GC that during these high CPU events the JVM is actively GCing because it is running low on memory. Obviously the memory settings need to be changed here but I wonder if we spent some time profiling the app and reducing its footprint if we wouldn't have to? I guess it depends if they will take the time to work on this effort.

JDBC driver versions

2008-05-21T07:59:00.000-07:00

In helping one of my colleagues this week I've come across another common problem that should be audited by everyone running a J2EE app server; JDBC driver version.

Typical scenarios:
1. If you're application has been working and all of the sudden starts to have strange SQL errors it never had before
2. Your application works with intermittent SQL errors (unrelated to SQL statement bugs).
3. See a lot of StaleConnectionExceptions

Check the version of the database server and then the version of the JDBC driver (in WebSphere Application Server the JDBC driver version is printed in SystemOut.log during the app server startup sequence). Typically the DBA just updates the server without telling anyone. This means that a bunch of clients are backlevel on the JDBC driver. I don't know what it is that some database vendors do in their fixpacks but they often seem to break the protocol used by the previous client drivers. So... audit your JDBC drivers periodically and if you can get your DBA to notify you of when updates are going on to the database servers you could even save yourself some grief in production.

Some common mistakes

2007-11-21T11:25:00.001-08:00

Sending the right files
I spent a frustrating day yesterday reading log files from a server that I was told had stopped running (i.e. the process was gone). I spent a long time reading the logs looking for an indication of a problem and couldn't find one. On examining the IP addresses in the logs it was clear they had sent me the logs from the wrong server. Sysadmins, make sure you send the right log files. Once I had the right log files I had the answer within the hour and a fix to go forward with.

Verbose GC
Too many people run with verbose GC turned off. Unfortunately the word verbose is over loaded and in this case verbose does not mean verbose in the Unix sense. Verbose GC is actually not that verbose and has a heck of a lot of great information in there. Run with the ST_VERIFY option if you are on pre-Java5 JVMs (i.e. earlier than WAS v6.1)

Stay current with maintenance
Time and time again I go in to debug a problem and what do I find? The problem is "known" and a fix has been available for some time. Just because things are working today doesn't mean that they won't break in the future. Yeah, I also understand the "if it ain't broke don't fix it" mentality and application servers are one place I don't like to play that card. Run the maintenance through all your test environments (including your performance test environment... you do have one, right?) and things should be okay in production. But don't expect to run on a 5 year old JVM and not run into problems one day...

I'm sure I'll think of more of my pet peeves so I'll add them later. In the meantime, if you're bored and need something to read see my paper on large topologies.

HTTP tuning

2007-11-19T07:50:00.001-08:00

I often have to modify the performance of the HTTP side of the conversation and here is a link to a decent basics:

Best Practices for Speeding Up Your Web Site

I also like the fact they used IBM's Page Detailer http://alphaworks.ibm.com/tech/pagedetailer which is an awesome application my colleague performance whiz Phil Theiller showed me. It is really awesome for getting a good visual feel for the HTTP side of things.

Lack of Performance Testing (load and stress)

2007-02-09T06:34:00.000-08:00

One thing I can not stress thoroughly enough is the necessity to conduct performance testing. I've already written an article on this subject (click here) and there is no reason not to do it. In fact, the #1 reason most critsits happen is either the complete lack of performance testing or very poorly executed testing. This is what I refer to as self-inflicted pain because, from my point of view, if you make the decision not to thoroughly test then you have accepted the reality that you will have a production outage and it will be difficult to resolve.

In addition, you should be monitoring your application using an appropriate tool like IBM Tivoli's ITCAMfWAS (IT Composite Application Manager for WebSphere Application Server). v6.1 was just released and I got a chance to play with it at a critsit over the past couple of weeks and it is a good improvement over v6.0 addressing some of the usability issues I had. It is tools like this that help me resolve a critsit in a matter of hours instead of days or weeks.

Day One SHAREABLE JDBC connection setting

2007-02-09T06:24:00.000-08:00

I do not know why I didn't start a blog on this topic earlier! This is what I do day in and day out so having an outlet where I can describe and discuss various scenarios is going to be helpful to me.

So, for our first topic we are going to talk about a common malfeasance in production performance that typically negatively affects any application that runs in the WebSphere Application Server environment. Okay, maybe not all that common but I've seen it three times now and obviously someone needs to start paying attention.

Inside the application EAR file there are deployment descriptors for the JDBC connection pool that refer to SHAREABLE and UNSHAREABLE connections. While there is plenty of documentation on this particular deployment descriptor there seems to be little understanding of how this particular setting works. BTW the default setting is SHAREABLE which is where most problems seem to stem from.

Now, what I'm writing here is not gospel so you will need to understand your application and how it uses JDBC connections particularly if there are transactional UOW involved. However, for the other 90% of you that are doing simple SQL queries and a few inserts here and there you will want to pay close attention... change the descriptor to UNSHAREABLE and run your performance test suite (you do have a performance load test, right?) and notice the change in behaviour of the JDBC connection pools. If you did this right you should see fewer connections being used.

This is because the default setting of SHAREABLE does not return the connection to the pool when you call close(). In fact, that connection is held with the thread until the thread finishes processing the request. This is an optimization that some people may need but the majority do not.

So the next question people always ask me is "why is this the default setting?" Well, heck, someone had to decide on what to use as a default and they picked one. But just because something is a default setting does not necessarily mean it is the right setting for your application. I think the JVM defaults to 128M for a maximum and you can pretty sure that probably is not the right maximum for 99% of the J2EE-based applications out there!