Ship on day one

The Importance of Setting Goals for Your DevOps Rollout

May 7, 2018

The one-time Google site reliability lead and current consultant will discuss the connection between product and organizational goals.

Typically if Carla Geisser shows up at your organization, it’s because things have gone sideways. (Her self-applied title, Software Exorcist, says it all.) But attendees of the Get Ship Done! Workshop will have the chance to learn from her in a much less stressful environment: The one-time Google site reliability lead and current consultant who specializes in building sustainable software organizations using SRE and DevOps philosophies will discuss how to use product goals to determine organizational goals. We phoned her up in New York to get a preview.  

What exactly is a software exorcist?
At Google I tended to work on systems that were a little old and often unmaintained. And there was usually a team of engineers who were avoiding them as if they were haunted. I ended up figuring out how to remove whatever the source of the haunting was and turn those systems back into things that were useful and maintainable. And now that’s what I do in my private consulting practice.

So when someone calls you, it’s because something has gone really wrong?
Often it’s because something’s gone wrong. I was involved the second year of fixing healthcare.gov, and that was clearly a thing that had gone wrong. But I also get involved when teams and software systems get stuck. They’ve basically ground to a halt because there’s some part of the system that everyone is afraid of changing.

 

What would you say to an engineering lead who says, “I’m totally ready to do this DevOps thing, but I just can’t get the executives on board with the investment”?
There are typically two reasons organizations I work with want to implement DevOps: Either they want to improve communications between the development and operations teams, or they want to increase speed. And the way you sell an organization on the speed benefits is by using the methodology in a small corner of your organization, proving that it works, and then using that success as a prototype for scaling it across the organization.

Start small and use that as a proof of concept.
Yes. And that was a huge thing we saw when I was working on healthcare.gov. There was one team that would show up to the weekly release management meeting and say, “We can do a rollout in five minutes. And if anything breaks, automated testing will catch it, and it will be back to normal in probably another five minutes.” Just by having that one team in the room changed the dynamic of the discussion. Management would look at the other teams and say, “Why can’t you do it like them?”

How do you bridge that gap when you have multiple teams with wildly different approaches to the work?
One big help in any of these cases is to just set engineering standards that include basic automation, testing, and monitoring. And then you can build those things into the organization itself. We talk about source control, we talk about having a database with issues and work items that’s shared by everyone, and you can do the same thing with your expectations for how well your software will operate. “Of course it’s automated. You’re not allowed to write software here if it’s not automated.”

What do you say to the engineering lead who says, “Hey, I hear what you’re saying, but I’ve got this legacy system that’s going to be impossible to turn around. Thanks, but no thanks.”
That’s literally my specialty. My pitch is usually to start very small, with the pieces that you have to change. Let’s say you have a large legacy system that’s mostly stable and you don’t want to touch. You should leave it alone. But if there are parts of it that you do need to change to update a feature or fix a bug, then you start by adding tests and building up as much automation as you can in that one corner of your legacy system.

What’s the biggest mistake an engineering team can make when trying to implement DevOps?
The biggest one I frequently see is organizations trying to do too much, too fast. The other is that people will take whatever words they used to describe their previous system, from job titles to internal documentation, slap the word “DevOps” on it, and say, “Okay, we’re doing DevOps!”

Do people actually do that?
Yes! I’ve seen it.

What do they think they’re accomplishing?
It’s not entirely clear. I think the people who are doing the job realize that you showed up one week and they had changed your title from SysAdmin to DevOps  Engineer, but you were still doing exactly the same job. The same thing happened ten years ago with the whole agile software development fad, where a bunch of organizations would basically put the word “agile” on top of their preexisting slow-moving development process and say, “Look, we’re doing agile now!” Nope, not really.

Okay, so what’s the number one thing you want to do set yourself up for DevOps success?
Can I give you two? As far as tactical changes, you want good visibility into how your system behaves. No matter what kind of system you have — legacy, a mixture of legacy and new, all new — you should know what it’s doing, how users are interacting with it, when they are being successful, and when they are failing. If you’re making changes with no monitoring, then you don’t know whether you improved things or made things worse.

You need a benchmark.
You need a benchmark. And then the other big one is making the process of writing code, checking it in, running it through whatever test suite, and then potentially deploying it into a production or staging environment is as straightforward, boring, and automated as possible. Because that takes a bunch of angst about delivering software out of the picture, and then your developers can then focus on whatever the product goals are.

Related Content