We've been designing software with this idea in mind for over 10 years. When you design with the idea of "everything fails, deal with it", your software becomes much more robust. Not only does it become more robust, but the maintenance becomes easier as well. Need to take down a server? Who cares, just do it, the system will respond properly. Database died? Who cares, just go back to sleep and deal with it at a reasonable hour. There is a downside however. You must make sure you have proper monitoring systems in place so that you will know when something has failed unexpectedly. Otherwise you could have your system running along in a potentially degraded state and you're not aware of it. It definitely takes all the stress out of managing a large system.