Ship on day one

Why You Can’t Have Too Much Validation in Your Dev Pipeline

May 10, 2018

The Oracle “cloud herder” says you should always be targeting 100 percent validation, even though you’ll probably never get there.

If there’s one thing Mark Mucha has learned after working at organizations as diverse as Lockheed-Martin, Amazon, and Oracle (where he’s currently “herding clouds”), it’s that there’s no correct way to do validation. That said, when he leads a discussion on the topic at the Get Ship Done! Workshop he’ll cover some aspects that are universal. He took a few minutes out of his day earlier this week to give us a preview.

You’re going to be speaking about validation. What does that word mean to you?
There are so many phases of validation. There’s validation at the code level, the test level. And then you’ll do validation on the things that you’re going to be running on in production. You’ll do some pre-production validation that might be communicating with production services. You should be validating your production stack even after you deploy, to make sure that you haven’t missed something, to make sure that you don’t have some deferred or delayed issue that was recently rolled out.


How do you figure out what needs to be validated?
The answer is probably that you need to validate a lot more than you’ll start out validating. You’re always going to target 100 percent validation, even though you’ll probably never hit 100 percent. You’ll probably never even know 100 percent of the things that need to be validated.

You’ve worked a lot with DevOps in the cloud. How does it differ from doing it in a more traditional environment?
The hosts that you’re running on aren’t as reliable as if you’d have physical hardware. But you also gain some flexibility and agility by running in the cloud. So there’s a bit of a tradeoff that you have to make. People that are coming from classic computing, where they’re running in their own company’s data centers, they can still do DevOps. But it would be a slightly different flavor of DevOps than if you were running in the cloud.

Obviously automation is the cornerstone of DevOps. What’s one of the most important things that engineering teams need
You have to build things into the software to support automation more easily. And in every organization I’ve worked in it typically doesn’t work that way. Typically, the things that would enable automation or make automation easier aren’t added in design. But this is also an example of where I think DevOps improves software development: The engineers who wrote the system are probably doing the automation for the system. The feedback loop is so much shorter, so they’ll learn quickly what they need to add to their software to better enable automation next time.

Of course, that’s assuming they can get the investment in automation. What kind of advice can you give the team leader who wants to lobby for more automation? It’s not always an easy sell.
You have to put in a little effort to come up with some tangible benefit that management is going to get out of it. Most likely, in most automation scenarios, you might be able to sell management on higher quality or more time to work on new features. But typically the estimates for the actual improvement are hard to come up with. You have to tie the benefits of automation to something tangible, something your management cares about. And organization to organization, company to company, team to team, that’s going to be different. So engineers need to do a better job of listening, understanding the business, understanding what’s important to the business, and then making sure that automation is improving the business in some way. Automation should never be around just to improve the engineers’ lives, because the business is paying for it. And I haven’t found a single company that’s interested in investing millions of dollars just to make engineers’ lives easier.

Can you give me an example?
Typically I’ll recommend to teams that they fight for something like a 10 or 20 percent budget that I would call a “small rocks” budget. And the small rocks can be the smallest building blocks for automation. I’d rather work on things that are going to have a wider impact that fixing a single bug, but the first thing to do is to make sure you’re doing automation that’s going to help the business. The second thing is, if you can’t get approval right away, do what you can to get started automating, especially if you’re automating yourself out of a painful situation. If you think you’ll have some kind of improvement due to the automation, you’ll also want to have metrics of some kind that shows how automation — even at the small rocks level — is improving things enough that you can justify a larger investment.

Related Content