tag:blogger.com,1999:blog-11670469915165972852024-02-07T03:58:59.062-08:00Alexandre Polozoff on WebSphere PerformanceThis blog is where I'll wax on about performance surrounding the WebSphere Application Server middle-tier integration environment.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.comBlogger82125tag:blogger.com,1999:blog-1167046991516597285.post-78248112121187432582011-03-30T07:03:00.000-07:002011-03-30T07:03:56.056-07:00Backup StrategiesI was having a discussion recently with one of my colleagues around server backups. He likened it to the spare tire of the car. Yes, you can drive around without the spare tire but does anyone want to do that for long stretches of time? Probably not.<br />
<br />
Servers need to be backed up. Simply if a server completely tipped over it can be duplicated, rebuilt and put back into service.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-31910923777693274152011-02-07T12:32:00.001-08:002011-02-07T12:32:53.274-08:00tprof switchesI wanted to capture this before I forgot about it ... <a href="http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp?topic=/com.ibm.java.doc.igaa/_1vg00014557b090-11cd67f58fb-7fe3_1006.html">tprof switches</a>Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-20924027158758679582010-11-30T06:10:00.000-08:002010-11-30T06:10:23.356-08:00CognosLooking for information about Cognos? I was. My colleague in the UK, Richard Collins, was able to point me to some informational links about Cognos. <br />
<br />
For full details on this product, see its home page here: <a href="http://www-01.ibm.com/software/analytics/cognos/business-intelligence/">http://www-01.ibm.com/software/analytics/cognos/business-intelligence/</a><br />
<br />
However, your most useful starting point is probably going to be: <a href="http://www-01.ibm.com/support/docview.wss?uid=swg27014432">http://www-01.ibm.com/support/docview.wss?uid=swg27014432</a><br />
<br />
...and in particular, this one: <a href="http://www-01.ibm.com/support/docview.wss?uid=swg27019126">http://www-01.ibm.com/support/docview.wss?uid=swg27019126</a>Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com1tag:blogger.com,1999:blog-1167046991516597285.post-8366450250662148542010-11-09T17:49:00.000-08:002010-11-09T17:52:28.298-08:00global transaction sharingIn the developers resource references there is an attribute for global transaction sharing.The value should be Shareable if the application uses global transactions. But if one is using <a href="http://publib.boulder.ibm.com/infocenter/wasinfo/v5r1//index.jsp?topic=%2Fcom.ibm.websphere.zseries.doc%2Finfo%2Fzseries%2Fae%2Fcjta_loctran.html">LTC</a> then one does not need to share the connection. Change the res-sharing-scope to Unshareable as noted above. This eliminates a lot of contention for connection pool threads. For more details on LTC see the link above to understand how it works.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-15225758393466040052010-09-24T05:24:00.000-07:002010-09-24T05:24:11.292-07:00The importance of dynamic circuit breakersIt is rare for anyone to provide details behind the root cause of a production outage. <a href="http://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919">Facebook</a> put out a report about an outage they had. If you're into troubleshooting and problem determination it is an interesting read. It sounds like they could turn off the particular function but had to completely restart the environment to do it. This is why it is important to have circuit breakers that can be activated dynamically. <br />
<br />
One also wonders what infrastructural changes could be made in the environment to help? It sounds like the application logic continued to retry requests. This is why I'm not a fan of applications automatically retrying requests because when failure occurs the retries can quickly overwhelm the back-ends. A firewall could have at least help shut off the pipe to the database. Though the consequences to the application would have been no different and would still have required a restart since there seemed to be no way to dynamically shut off that particular function. <br />
<br />
Certainly the error logic sounded confusing at best. And error paths through code are the ones least frequently tested so they tend to fail magnificently in productionAlexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-32863427333634423832010-09-14T15:46:00.000-07:002010-09-14T15:46:58.394-07:00grep Exception SystemOut.log | wc -lI'm reminded today that after performance tests a simple check exists especially when adding more JVMs to a cluster. Count the number of exceptions in the log files. Of course, clear the logs before running the test. If the counts are not all roughly the same (or significantly skewed from the other app servers) then it is clear there are issues with that JVM that need to be checked. Sometimes it is configuration or a misplaced JAR file.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-41801221469987913212010-06-08T04:33:00.000-07:002010-06-08T04:33:55.989-07:00Been a busy 2010I know I haven't kept up with the blog this year. While the blog post I'm <a href="http://dilbert.com/blog/entry/the_value_of_ideas/">linking to today makes no mention of performance</a> it has everything to do with performance. Maybe one day I'll get a chance to sit down and explain my thinking. Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-79878854949416888562009-12-24T06:53:00.000-08:002009-12-24T06:53:28.254-08:00Catching up on closing up 2009It has been a busy few weeks for me which is one reason I haven't posted in a while. Some things I've been thinking about lately revolve around high availability, switches and logical partitioning of applications and application servers. I'll try to post more on these subjects over the next few weeks. <br />
<br />
Hope you all have a great holiday season and a Happy New Year. I'll see you on the other side of 2010!Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-80214336322687248382009-11-10T09:18:00.001-08:002009-11-10T10:14:37.319-08:00The people side of performanceThose of you that read my blog know that I tend to primarily address technical topics when it comes to performance. The latest issue of Communications from the ACM (10/2009 vol 52 No 10) has an article called "The Business of Software Contagious Craziness, Spreading Sanity" by Philip G Armour. I was intrigued with some of the points Philip brought up in his article. Particularly this tidbit<br /><br />"... having a strong conviction that something cannot be done is usually a self-fulfilling prophecy. If people are convinced that something is not achievable, then they usually won't achieve it - if we argue for our limitations, we get to keep them." <br /><br />Wow! Nail on the head. So how do we convince an organization that something that is currently perceived as impossible actually is quite achievable? <br /><br />First off, people have be encouraged to think outside the box. George Patton is quoted as saying "If everyone is thinking alike, someone isn't thinking." Coming from the battlefield I think we can understand why the General said this. He needed his commanders to not only visualize the current plan of operations but also what alternatives and/or competing strategies were available. One of my friends, Keys Botzum who also happens to be an IBM STSM, once put it this way. "The patient is dying on the table. What can we do, right now, to keep him from dying." It is a different perspective for techies to mull over.<br /><br />Likewise, a technical team can not be lead by consensus. Technical teams have be be lead by dictatorship. Someone, at the top, needs to make technical decisions and take responsibility for those decisions. Right or wrong someone, like General Patton, has to tell the troops which way to charge. He can take input from his Lieutenants however the decision is ultimately his and his alone to ensure it is the correct decision. Which is why the other side, led by mad men who knew nothing about how to fight a war, lost. The General knew what he was doing and knew how to lead his people to victory. <br /><br />Which brings us to our next topic. In order to be able to lead a technical team the leader has to be technical. I understand management's philosophy that they are managing people and not the technology. The problem is that technology, by its very nature, is complex. This is simply unavoidable. We're not managing a bunch of lawyers (sorry to pick on lawyers, you're used to it by now, as they are a good example). What lawyers do is not very hard to comprehend. They sue people. They defend people. They argue in court for their client. They draw up contracts, etc. None of this is rocket science. Start running multiple servers in parallel handling thousands of transactions per second and that organization is starting to approach NASA-like technical complexity issues. In order for a technical team to succeed they must be lead by a strong technical person. It is unreasonable to expect someone who does not comprehend the subtleties of technical subjects such as "verbose GC" or "JDBC deadlocks" to be capable of making a technical decision much less the correct one. <br /><br />In conclusion, and back to the "self-fulling prophecy" and "that something is not achievable" that started this line of thinking for me. There are two ways to avoid failure. One is to have a strong technical leader who can take the reins as the technical General who makes the decisions. The other is for executive management to support the technical leaders. This means not only in terms of vocal agreement but also in terms of financial dollars to the project, making people available to do the tasks and any other strings that need to be pulled in order to make things happen. <br /><br />I'll have more to say on this subject later as this is about all I can muster in the limited time I had for this "lunch & learn" session.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-4390990896709247592009-11-05T07:52:00.000-08:002009-11-05T09:26:33.140-08:00Managing the application threadsThreads and multi-threaded programming is not a trivial subject. But some application developers find themselves having to start their own threads as part of their application. <br /><br />I'll start this post off saying that using unmanaged threads is never a good thing. When using threads developers should be looking at how to manage all their threads in their application. Never exit the application without terminating all of your threads. Why would we need to do this? <br /><br />An interesting problem is when an application is stopped (not the JVM/app-server but just the application), a new version is deployed, but the restart fails because there are still threads hanging around from the previous version of the applications and older versions of the same classes. I never would have imagined this problem occurring but I can see that if the application lost track of its threads that it would not be able to close them when it shut down. <br /><br />Java provides the capability to manage all the application threads by using thread groups. When creating a thread and assigning the application's thread to a thread group all the application has to do is capture the "application stop event" and use the stop method of the thread group thereby stopping all the threads in the thread group. <br /><br />The beauty of this solution is that the application can reliably manage all of the threads it creates though a single interface. It is also portable which means your application will work with any application server.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com1tag:blogger.com,1999:blog-1167046991516597285.post-36683966779930051062009-11-05T06:06:00.000-08:002009-11-05T06:11:15.202-08:00High availability through parallel cellsHigh availability sites have unique requirements that are not easy to solve. However, one solution does revolve around using multiple cells in production. This allows for easily taking a cell out of rotation to make changes. Then when the cell goes live and if the changes didn't work it is just as easy to take the cell out of rotation again. I've used this strategy at many, many 24 x 7 sites. <br /><br />Here is the link to <a href="http://www.ibm.com/developerworks/websphere/techjournal/0911_col_polozoff/0911_col_polozoff.html?ca=drs-">my latest article</a> on multiple cells.<br /><br />The only problem that arises is if things like database schemas change in the backend which also means application code changes. Those kinds of changes require more work and possibly parallel schemas until the transition is complete.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-16702178374097945662009-10-19T13:50:00.000-07:002009-10-19T13:58:40.289-07:00Running out of disk spaceAn event occurred this morning that reminds me that performance problems are sometimes environmentally driven. One application went down with various errors related to the file system. After some investigation it was discovered that the /tmp space ran out of space. Yet no one knew it had happened. Turns out that no monitors are in place to watch for resource exhaustion like low disk space. <br /><br />One could argue why did /tmp fill out if daily (automated) housekeeping tasks were in place (they were not). But that is a pointless argument if someone moved a whole bunch of files in /tmp. They could have moved files in there to install them and then delete when the install is done. Or they could have been log files being transferred to another environment via ftp. In any case, an automated monitoring tool should have alerted the operators that disk space has come close to exhaustion and need to take action to remedy the situation.<br /><br />If automated monitoring is not in place then the production applications that make use of /tmp space will fail causing an outage. And without alerts about low disk space it can take several hours for the operations and troubleshooting teams to figure out that the application failed because it could not write out a temporary file. A several hour outage that could have been completely avoided had the right resource monitors been in place. <br /><br />Other resource monitors should be looking at CPU utilization, page file activity, etc.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-55738104432721917362009-09-30T08:36:00.000-07:002009-09-30T08:42:58.575-07:00IPv6Last year I <a href="http://websphere-performance.blogspot.com/2008/08/one-reason-i-like-to-take-thread-dumps.html">blogged about handling ipv6 lookups </a>that caused threads to wait thus slowing them down. Not surprisingly I've come across this a few times since then. What is interesting is the effect it can have on throughput and response time. Testing showed that once <tt>-Djava.net.preferIPv4Stack=true </tt>was enabled the test had 8% improvements in both throughput and response time. This is truly interesting data. That is a big difference and a good reason to periodically take thread dumps even if everything seems to be running nominally. One never knows when they may uncover a gem of a performance boost like this one.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-54430548790048086592009-09-24T09:34:00.000-07:002009-09-24T14:52:38.348-07:00PMI = all is not a good setting for productionI am reminded occasionally when debugging production issues that setting the PMI level in WebSphere Application Server to "all" is not a good thing to do. At the "all" setting one can see an application exhibit negative behavior. I am also learning no two applications will always exhibit the same problem. Some applications crash and burn under the weight of PMI=all incapable of providing a response to any request. Other applications seem to continue to function nominally but the CPU for those processes seem to be considerably higher.<br /><br />Finally, regardless of the PMI settings configured in the production environment it is imperative that all members of a cluster are set to the same values. Having some processes set to one level and other processes set to another results in higher CPU utilization from some JVMs and not others.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com1tag:blogger.com,1999:blog-1167046991516597285.post-56000545664632974222009-09-23T20:34:00.001-07:002009-09-23T20:35:34.021-07:001 minute garbage collection cyclesDoes your application use RMI? Are you seeing garbage collection cycles with exclusiveaccess every minute (60,000 ms)? You may need to apply the following two parameters to the JVM command line. <br /><br />-Dsun.rmi.dgc.client.gcInterval=360000000 -Dsun.rmi.dgc.server.gcInterval=360000000Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-18115874444208205672009-09-22T17:13:00.000-07:002009-09-22T17:16:16.695-07:00Essential log attributes to log - duration of a request in access.logIt is absolutely imperative that the access.log file from your Web server record the total response time as measured by the Web server. This is available in all flavours of Web servers. It provides definitive response time numbers as they leave the Web server that can be used as a second data point to the application monitor's servlet response time. The log entry also allows sysadmins a chance to put their log monitors on it to alert on long response times.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-28353131693374116872009-09-22T16:56:00.000-07:002009-09-22T17:12:01.485-07:00Default values - they're not for everyoneI am frequently asked about about the significance, or lack thereof, of default values to one or more configuration items. People need to remember that default values are simply starting points so the environment can be brought up. For example, an operating system has the default value for TCP Keep-Alives set to 2 hours. According to <a href="http://www.faqs.org/rfcs/rfc1122.html">RFC 1122</a> this is an acceptable default value. However, if you look at the acceptable ranges of values it starts at 10 seconds and can be set as high as 10 days. So, obviously, the default value is not going to work for everyone. Some sites may need to set it low to around 10-15 seconds. Other sites might need a 2 or 3 day setting.<br /><br />Additionally a comment was made in general about setting the value of TCP Keep-Alives can not be lower than 2 hours. I think it was a misreading of the RFC specification that reads "This interval MUST be configurable and MUST default to no less than two hours." Read that the DEFAULT must not be less than two hours. This does not imply that the value can not be lower than two hours. The lesson learned there is to read the specifications to the letter. Unfortunately I think the emphasis in the RFC on the word "MUST" does distract the reader from the word that follows "default" and can subtly mislead the reader. Erroneous information is consequently passed on as a rule. Unfortunately, if no one backtracks reads the RFC and verifies the rule then rampant disinformation is spread and becomes written in stone.<br /><br />Trust but verify every setting in your environment. Every setting should be tested as thoroughly as possible. Documentation should then record what was tested and what values were selected over others and why. This documentation will be valuable to the group of people maintaining the application 5 years from now and I can guarantee it will not be you the reader. It will be whoever is watching the store after you have moved on to new and exciting career opportunities.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-52443820288301997002009-09-16T08:31:00.001-07:002009-09-16T08:32:25.907-07:00Enable verbose GChttp://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21114927<br /><br />I really recommend running in production with verbose GC enabled. The use of the term verbose is unfortunate as it is not as verbose as people think it is. And the data collected is invaluable. <br /><br />If you have any doubts, enable verbose GC in test and see if you can measure any impact from enabling it.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-4241077106519934472009-09-16T08:11:00.000-07:002009-09-16T08:17:10.927-07:00socketread0 timeout on JDBC connectionsSince I run into this problem every once in a while I'm putting up a note here to help others when they run into this problem. JDBC calls can sometimes get stuck on socket read calls to the database if some rather nasty network problems exist. The only way to determine if network problems exist is to use tcpdump (AIX, Linux) or snoop (Solaris) to capture the packets. One can then use Wireshark (or its predecessor Ethereal) to read the capture files. If you see issues like "unreassembled packets", "lost segments", "duplicate ACK" or checksum errors then most likely the network is having some abhorrent behaviour affecting the server. If random threads hang on socketRead0 calls that never seem to get a response then the only way to deal with this is through timeouts. <br /><br />On DB2 follow use this parameter:<br /><br />http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=/com.ibm.db2.luw.apdv.java.doc/doc/r0052038.html<br /><br />blockingReadConnectionTimeout<br />The amount of time in seconds before a connection socket read times out. This property applies only to IBM Data Server Driver for JDBC and SQLJ type 4 connectivity, and affects all requests that are sent to the data source after a connection is successfully established. The default is 0. A value of 0 means that there is no timeout.<br /><br />For Oracle use:<br /><br />oracle.jdbc.ReadTimeout<br /><br />http://forums.oracle.com/forums/thread.jspa?messageID=2326985Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-20149653890152011572009-08-18T21:15:00.001-07:002009-08-18T21:21:59.265-07:00Logs, log identification, log monitoringLogs are an ultimate source of information to the health of an application. Logging data helps understand if something has gone wrong and what might have gone wrong. The problem is that there are two sides to logging.<br /><br />Developers provide the first level which are the log statements themselves. I have always found that a major/minor log code (i.e. (error/warn/info/trace = major; minor = module/method/logic) to be the most straight forward way to provide log information. Identify the type of log message being recorded to a specific category.<br /><br />For the operations side of the house they can monitor logs and based off of known major log codes they can decide if they need to take action. A major code that indicates an error can be immediately dispatched to a field technician. While other levels can be monitored or collected if in troubleshooting mode. <br /><br />However, if an application development team does not provide any well defined codes for log messages and only provide free form text then the operations/runtime side has difficulty determining if a specific log entry is a problem or not. Unfortunately, most application development teams choose the free form text logging as their solution which makes runtime operations that much more difficult and, in many cases, confusing.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com1tag:blogger.com,1999:blog-1167046991516597285.post-22400758388375816542009-07-28T06:37:00.000-07:002009-07-28T06:56:53.559-07:00Keeping notes (AKA your operational run book)This morning I was reminded about the importance of keeping notes about the problems I encounter and the solution that provided the fix. A co-worker of mine researching a problem came across my <a href="http://websphere-performance.blogspot.com/2008/08/one-reason-i-like-to-take-thread-dumps.html">blog post</a> through a google search. It seems their environment is suffering from the same symptoms. Had I not kept a note about that issue they would have most likely gone through the same level of effort I originally did to research and find the solution. Hopefully when they apply the fix it resolves their problem (I think it will) and they didn't have to spend several hours/days to get to resolution.<br /><br />This brought to mind the importance of a run book. If you are not familiar with the term a run book is basically an operations manual. It spells out what needs to be done for various tasks. For example, if we need to deploy a new application into production the run book has a full set of instructions (a recipe if you will) on what to do. Likewise, when the trouble shooter debugs a problem they record in the run book what the problem was, the steps they took to determine root cause, the fixes they tried and which one(s) finally worked. This way should the problem reoccur and a different shift of people see the same problem they can refer to the run book and go through the same steps. <br /><br />The run book does not have to only address technical details. It can also provide operational response instructions like how to run a war room,who needs to be involved, when various organizations/teams are engaged, what triggers an engagement, etc, etc. Every operational detail can be recorded in the run book. <br /><br />Does your production environment have a run book?Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-37885333545900368342009-07-20T08:33:00.000-07:002009-07-20T08:41:48.329-07:00Finding information (like tuning guides)I frequently get queries about finding information on one topic or another (generally around performance or application monitoring). A co-worker in Australia was looking for some general WebSphere Web services tuning guides. IBM.com has a wealth of information on a variety of topics so I thought I would put together this post on how to search for information.<br /><br />This particular <a href="http://www.google.com/search?q=site%3Aibm.com+web+services+tuning&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a">google search</a> is one I used to find information for Web services tuning.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkd4jeGY91iaCoMb_nRLv3JXeM75R-xJsGTw3N_lpAuVUakKZDcS7o1113U1TKxvLUyymO_5RSJEkQw7TOsW7DnoYN-HIx3zE6ldz6zSP_zLIG_OS1ZdHHPFGZy0HAIN4gHeXdAaeE-g/s1600-h/google.search.jpg"><img style="cursor: pointer; width: 320px; height: 55px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkd4jeGY91iaCoMb_nRLv3JXeM75R-xJsGTw3N_lpAuVUakKZDcS7o1113U1TKxvLUyymO_5RSJEkQw7TOsW7DnoYN-HIx3zE6ldz6zSP_zLIG_OS1ZdHHPFGZy0HAIN4gHeXdAaeE-g/s320/google.search.jpg" alt="" id="BLOGGER_PHOTO_ID_5360567504101705250" border="0" /></a><br /><br />You'll note the key in the search string is the beginning "site:ibm.com" which restricts the search to only those items found at ibm.com.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-47628123107697248142009-07-20T08:31:00.001-07:002009-07-20T08:32:52.957-07:00Latest articleI forgot to post when my <a href="http://www.ibm.com/developerworks/webservices/library/ws-probdetermination/index.html?S_TACT=105AGX04&S_CMP=EDU">dW article came out</a>. This is a new series I'm going to write on defensive architectures. Part 2 is being reviewed by my colleagues right now.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-48525637901284062272009-07-09T12:37:00.000-07:002009-07-09T12:44:19.454-07:00Problem Determination - High CPU strategies on Windows OSI've been quiet the past couple of months but that is because I have an article coming out on deveoperWorks soon. I'll post a link to that when it is ready. <br /><br />A colleague called me today to talk about a high CPU scenario and steps to take to try and resolve it. My first recommendation was to get thread dumps of the JVM when it hits the high CPU scenario. Then I found out the JVM is running on a Windows OS. Since it is not possible to execute a kill -3 on Windows and one has to use the script methodology with no CPU available it is tough to get a thread dump. <br /><br />I suggested he collect thread dumps as the CPU starts to climb. This works if they can (a) predict when the CPU will start to climb and (b) it climbs slowly enough to be able to collect javacores along the trajectory. It sounds like the CPU can spike in a matter of seconds even at stead load volumes. Ugh.<br /><br />My final thought is to reduce the thread pool max in half and rerun the test. Perhaps there are simply too many threads executing and nothing will prevent it from going 100% CPU. Though I have seen code that can drive even an 8-way to 100% with just a few threads. I'm waiting to hear back. But I think the latter strategy of reducing the thread pool max will be the best course of action for them. I expect an email in the morning.Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0tag:blogger.com,1999:blog-1167046991516597285.post-41409679999160184552009-03-26T15:13:00.001-07:002009-03-27T05:39:24.434-07:00Predicting Capacity<div xmlns="http://www.w3.org/1999/xhtml">I have had a number of calls around "We have an application, not built yet, but we want to know if we have enough capacity and/or will the application scale/perform in our environment."<br /><br />My analogy to this is the following. Someone shows me the blueprint to a Formula 1 (or NASCAR) race car. They ask me "will this car win the race?" <br /><br />While the car being proposed may have all the right components (wheels, engine, gearbox, etc) there are other unknown variables that determine if the car can win a race. Without extensive testing there's no way to know if all the components are assembled correctly nor if they are configured to work together in optimal manner (performance tuned). <br /><br />Related to this is the ability of the car to both get off the starting line and finish the race (a prerequisite for winning). Again, there are a number of variables that impact this. First is some catastrophic failure in one of the components. Then there is the skill of the car's driver. Then the other drivers on the same racetrack and crashing into our car. <br /><br />There is simply no way to predict if the car can win much less finish a race. <br /><br />The same attitude has to be taken when planning your production IT environment. You can't predict behaviour. You can, on the other hand, conduct stringent system integration and performance testing in order to see if your application will (or will not) perform as expected in production. <br /><br />This is why everyone should test as early in the development cycle as possible. I was working with one application that had been in production for several months with constant, recurring failures. There were fundamental application design problems in how the application was developed. It took us four more months to fix those design problems and get the final, new version of the application into production. Don't put off testing. Do it right. Do it early. Test early. Test often. That is the key to success in production. <br /><br /><div class="zemanta-pixie"><img src="http://img.zemanta.com/pixy.gif?x-id=2e64ccd4-2225-8fb7-812f-f0a3f9fbda08" class="zemanta-pixie-img" /></div></div>Alexhttp://www.blogger.com/profile/16310744996196312026noreply@blogger.com0