This is the kind of stuff that makes us admins infuriated at developers. Just to illustrate, pretend we have 3 testing environments and then Production for a web app. Env1, Env2, Env3, Prod. It is expected code will be moved up through those environments sequentially.
Developer Ted: I am rolling out code to environments Env1 and Env2 at the same time.
[a few hours later]
Developer Ralph: Env2 is broken and my coworkers and I can’t get anything done. What’s wrong?
Admin Mike: The code rolled out to multiple environments earlier broke things and made the environments unstable. We’re working to fix this now. Talk to Ted who rolled it out in multiple places all at once to not do that in the future.
Developer Ralph: But I need to work now, and so do my coworkers. We’re going to start doing our work in Env3.
Admin Mike: [blank look knowing I don’t have authority here]
[short amount of time passes]
Developer Ralph: I need support in Env3 because it is not working properly now.
Admin Mike: Well, some of the stuff you moved up shouldn’t have been moved up and that environment is borked now and we’ll have to expend more energy to fix it.
Developer Ralph: But I and my people need to work, should we start moving to test in Production?
At this point strangling the developer actually seems like a plausible mitigation to further destruction and downtime…
Developers should not have any access, not even sudo, not even an ssh/ftp login or capability of launching a remote debugger into anything production related. They should have less privileges than most regular users of Internet-facing web applications.
If you were using a continuous integration build server, this would be less of an issue. Try Luntbuild {professional}, Hudson, Bamboo, CruiseControl{.NET|.rb}, Tinderbox, Continuum, Pulse, Gauntlet, Anthill{Pro}, DamageControl, TeamCity, BuildForge, Draco.NET, ParaBuild, QuickBuild, Sin, Bitten, BuildBeat, BuildBot, CM Crossroads, Gump, PerfectBuild, Pragmatic Automation, DrumBeat, BeetleJuice, Smolder, Launchpad, QuickCheck, Litmus, etc. There isn’t a shortage of build server support out there.
For a general listing of these, be sure to check out:
http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix
I have come from mostly Java environments, where Luntbuild, Hudson, TeamCity, and Bamboo have some of the best integration capabilities from my direct experience.
In this situation, I do find it amazing that developers Ted and Ralph have jobs. Please tell me that they’re 15 years old and have zero experience with software engineering and testing.