Anytime one pushes out a new release candidate it should be tested in the same configuration as it will be in the production environment. This means that any shared libraries must be deployed out with the applications that are using them when multiple applications share the same app server.
The best that can happen is that the production environment will remain stable as long as what was tested is what is in production. However, the worse that can happen (and it will) is the site will become unstable. That is not the end of it though. Whatever deployment was put in place has to be backed out (here is where multiple cells really helps you out and if you don't know what I'm talking about go to this great article by my colleague Peter) and the previous deployment has to be re-established in production. The decision to revert also has to be made. This is, depending on the people running production, a difficult decision to make. Even more difficult if a back out cell is not available.
Therefore, depending on when it is discovered that the production site is unstable and the decision to back out is finalized it can be several hours before production can be stabilized.
This is one reason why it is important to test prior to deployment in production and that "testing in production" is never a good idea.
No comments:
Post a Comment