Once you understand what DevOps is about and the benefits it can have on your organization, the next question is how do you implement it?
The first reaction for most people is to research the toolsets. Unfortunately, it takes more than purchasing a set of software tools and telling your development and operations teams to collaborate more. DevOps isn’t something you can buy. It has to be incorporated into your organization and grown over time.
The core to DevOps is the CAMS approach. The term CAMS was coined by Damon Edwards and John Philips in Silicon Valley back in the 90s and early 2000s, as a way to define the culture of empathy.
DevOps starts with learning how to work differently. It embraces cross-functional teams with openness, transparency, and respect as pillars.
The CAMS mindset is not compartmentalized—a trap many organizations fall into by rewarding individual achievements at the expense of the team. Rather than two goals, one for development and one for operations, there is a unified goal of rapidly deploying code safely and reliably. Achieving this takes a shift in culture, which takes a shift in team structure and process.
In order to fix what goes wrong, or repeat what went right, you need to have systems in place that can be repeated. It should be thought of like an assembly line. Although humans are involved, they shouldn’t be touching the code. The journey from development, through test stage, and into production should be completely handled in an automated way.
With that being said, you need testing and abort paths. If you commit something that fails to meet standards, it should get rejected. If it fails to stage, fails to deploy, then the code should not go any farther.
“If the build, deploy, test, and release process is not automated, it’s not repeatable.”
“When steps are taken manually, they are error-prone, and there is no way to review exactly what was done.” – Jez Humble and David Farley, Continuous Delivery.
The most expensive work is rework. Measurement is about monitoring and tracking performance, so there is feedback. Errors should be expected, but the goal is to learn once for the organization and not repeat mistakes.
Measurement is also about aligning incentives. When incentives for different team members work against each other, you end up with a lose-lose situation, low productivity, team frustration, and roadblocks. The entire team needs to be aligned to achieve the same outcome.
For example, one team was challenged to write a new software test. The result was that everyone wrote a test, checked the box – but nobody tested anything. The incentive was tied to writing, not testing, so the strategy failed. Incentives need to be tied to the right outcome.
To understand this better, you can look at game theory and outcomes. John Nash has devised a solution concept known as the Nash Equilibrium.
Using his system, we can determine when we have a Nash Equilibrium or a win-win situation. A broken traffic signal is a good example. Even if there are no police around, two drivers will likely choose to obey the law and stop, so neither driver crashes. One driver must allow the other to go. The other options are they both go and crash, or no one goes, and they remain stuck. Cooperation is required for the best outcome. Either one driver will decide to allow the other to go, or they will go and expect the other driver to wait.
Sharing requires nurturing a blame-free environment. People are adept at hiding problems when they think they’ll be punished or blamed for their mistakes. Punishment doesn’t help them make fewer mistakes – and, on top of that, you have no idea what’s going on in your environment or how to increase employee productivity.
Matt Stratton, Senior Solutions Architect at Chef, had a customer who developed what’s known as the “blameless post-mortem.”
“The thing is, people are always going to make mistakes. You can’t fix that. People are human. So stop trying to fix them. Instead, we fix systems. That’s what we mean when we say we have a blameless post-mortem,” Matt said.
“For example, if a website goes down, we fix it. We don’t sit there and say it’s because John or Jane deleted these files. We identify the slip, but we don’t sit there and say it’s because you messed up. We’ll punish you so you don’t do it again.
Instead, we say—how was John or Jane even able to do that? Then, we make the system more resilient. Then, when a human makes a mistake, it doesn’t crucially compromise the system.”
The upside of the blameless post-mortem is the opportunity for each member of the team to weigh in on what went wrong. You can focus on identifying the problem rather than claiming immunity.
Where can we automate better? How can the team as a whole act to improve? How much quicker can we turn around and get the product into the customer’s hands?
A blameless culture is a culture of sharing.
The ultimate goal of a DevOps team is to get new software into the hands of customers as quickly as the company strategy intends. Achieving this is achieving continuous delivery.
The only way to achieve continuous delivery is to follow the CAMS formula.
Yes, there are tools needed to automate, measure, and share feedback. However, the first step in succeeding with DevOps is the culture – creating a cross-functional team guided by empathy, transparency, respect, and alignment to produce the same outcome.