The Self-Fulfilling Prophecy of Bad Release Processes  ⚐

02 Feb 2012

If you need to wait any non-trivial amount of time between completing something and seeing how well it’s
performing, you’re not going to be working on that project by the time you get your answer. When you do get your
answer, you’re not only going to have to refresh your memory on what you had been working on, but you’re going
to have to do the same on whatever else you had started working on.[1]

I will admit, I’m currently a bit frustrated with our team’s development and release process. In the name of stability, we have given up speed, agility, and, honestly, stability.

Ironic, eh?

When you’re working on web stuff, having a long cycle between when something leaves your keyboard and when it is live on production servers being poked at by real, live customers is a bad idea for a whole bunch of reasons.

There’s always something different about production than your staging environment. It doesn’t matter how well prepare you, it just happens. Different security, some small schema change, a firewall rule. Something. The longer you wait between the last time you worked on your code and when it is in front of customers, the more likely something else will change, too. And the less likely you’ll remember how your stuff worked.

Somehow, handing over code to the QA team is considered to be a sanctified handoff. “My part is done. I’ve given the code to QA.” The fact that it doesn’t quite work, has significant issues, or blows up is irrelevant. The division of labor between developing and deploying makes it far too easy to pass the buck and make the brokenness someone else’s problem.

And, I think worst of all, you take your eye of the ball. If you’re engaged in your release, feel responsible for it, you can do smart things like push it live, make sure it’s working, take a look at some logs to see if your assumptions are right. Maybe you speculated on some language or colors, and it’s just not converting. Change it.

When you’ve decoupled your development and release processes, you don’t have that fine grained control, that immediate responsiveness. Instead, you need to get someone off of another project, have them dive back into the code, put it through another release process, and repeat it all over again. A change you could have made in five minutes and might have benefited your company or customers ends up in another week long QA process where, inevitably, some inexperienced QA person digs up some completely unrelated bug, and your watch your life just waste away …

Yes, long, rigid release cycles are necessary for software products that actually have to ship code, or might destroy data, or blow away finances, or do really bad things™

But if you’re working on the web, all they do is make things worse. I sincerely believe that. They don’t make things more stable (too many other things changing; too much context switching to remember why you had that odd bit of code), or make your releases smoother (too much passing of the buck; too many people involved; too many places where someone can say “stop the presses” because their nose is out of joint that day). They just make people feel better, and they let people cover their asses, so that the self-inflicted pain of choosing that release style can’t possibly be blamed on them. It’s a self-fulfilling prophecy. By making the process to allow people to cover their asses, you ensure that they have something to cover them about.