Getting to CD In An Org Predisposed To Hate It  ⚐

23 May 2017

I thought this presentation by Dan McKinley was really interesting and resonated heavily with my experience in helping to shepherd an organization that was pendulum swinging from everybody hacking production, to nobody getting to do releases until you filled out a form in triplicate, to an org that was doing 8–10 releases on most days.

We never got to continuous delivery (CD), for a bunch of reasons, but mostly:

• Cultural (it scares the crap out of the systems and support teams, even if it might be better for them)
• Technical (it requires good tests and good dev/beta systems, and we’ve always been underinvested in the resources to help there)
• Organizational (we’ve rarely settled into a structure that allowed our teams to develop the discipline)

But we did continually get better, and I’m guessing in another year or so, with the right people pushing, I don’t think a real CI (continuous integration)/CD pipeline is unreachable.

Some bits from the presentation that were particularly resonant with me …

Namely, we had a lot of process that was prophylactic. It was built with the intent of finding production problems before product

As your organization gets bigger (and not even, like, really big, but just bigger), there are lots of people who think their job is to protect the production org by creating all sorts of process to make it really hard to get something to production. In reality, all that process just makes people pay less attention, not more attention. There’s always somebody else who is more responsible for the code going live, being tested, being right. The further away you are from being on the hook, it’s natural that you pay less attention.

Which is why, smaller, more frequent releases, with less friction and less overhead, makes a lot of sense. It’s your responsibility to make sure you don’t break production, and if you’re going to be responsible, don’t you want to make smaller bets? That leads to this tenet …

Deploying code in smaller and smaller pieces is another way. In abstract, every single line of code you deploy has some probability of breaking the site. So if you deploy a lot of lines of code at once, you’re just going break the site.

And you stand a better chance of inspecting code for correctness the less of it there is.

There’s a lot of goodness in this presentation, resulting from the scars of helping to drag an engineering team into something that works, that has buy in, and increases the velocity and performance of the team (and helps keep everybody happy because they’re working on stuff that actually gets to production). There’s some bits towards the end of the presentation that make sense for one big team, but less sense for multiple teams. Multiple teams is a huge way to help solve this problem. If you can break up your application into smaller, separate applications, or services, or microservices, or trendy term du jour, then you can reduce your dependencies between teams.

That lets each team reduce it’s risk and some teams can ship 50 times a day, and some 10, and some 2. It increases a bit of coordination between teams, but with good documentation and smart API design (ideally with good versioning so that team releases don’t have to be coupled), you can get to a point where teams can all be really efficient and not beholden to the slowest of teams.

Anyway, it’s a long presentation, but I think it’s a really great, real world example of how to get a big challenging org into CD (or at least on the path to it).