Lately Gene Kim, Kevin Behr, and I have been on a nearly messianic crusade against IT suckage. Much of our discussion has centered around The Visible Ops Handbook that Gene and Kevin co-authored with George Spafford. Visible Ops is an extremely useful playbook containing four steps that IT groups can follow to help them become much higher performing organizations.
However, I will admit that Visible Ops is sometimes a hard sell. That’s because the first step of Visible Ops is to create a working change management process within the IT organization– with functional controls and real consequences for people who subvert the change management process. Aside from being a difficult task in the first place, just the mere concept of change management causes many IT folks to start looking for an exit. “We hate change management!” they say. “Don’t do this to us!” What I quickly try to explain to them is that they don’t hate change management, they just hate bad change management. And, unfortunately, bad change management is all they’ve experienced to date, so they don’t know there’s a better way.
What are some of the hallmarks of bad change management processes? See if any of these sound familiar to you:
1. Just a box-checking exercise: The problem here is usually that an organization has implemented change management only because their auditors told them they needed it. As a result, the process is completely disconnected from the actual operational work of IT in the organization. It’s simply an exercise in filling out and rubber-stamping whatever ridiculous forms are required to meet the letter of the auditors’ requirements. It does not add value or additional confidence to the process of making updates in the environment. Quite the contrary, it’s just extra work for an already over-loaded operations staff.
2. No enforcement: The IT environment has no controls in place to detect changes, much less unauthorized changes. If the process is already perceived as just a box-checking exercise and IT workers know that no alarms will be raised if they make a change without doing the paperwork, do you think they’ll actually follow the change management process? Visible Ops has a great story about an organization that implemented a change management process without controls. In the second month changes were down by 50%, and another 20% in month three, yet the organization was still in chaos and fighting with constant unplanned outages. When they finally implemented automated change controls, they discovered that the rate of changes was constant, it’s just the rate of paperwork that was declining.
3. No accountability: What does the organization do when they detect an unauthorized change? The typical scenario is when a very important member of the operations or development staff makes an unauthorized change that ends up causing a significant outage. Often this is where IT management fails their “gut check”– they fear angering this critical resource and so the perpetrator ends up getting at worst a slap on the wrist. Is it any wonder then that the rest of the organization realizes that management is not taking the change management process seriously and thus the entire process can be safely ignored without individual consequences?
I firmly believe that change management can actually help an organization get things done faster, rather than slower. Seems counter-intuitive, right? Let me give you some recommendations for improving your change management process and talk about why they make things better:
1. Ask the right questions: What systems, processes, and business units will be affected? During what window will the work be done? Has this change been coordinated with the affected business units and how has it been communicated? What is the detailed implementation plan for performing the change? How will the change be tested for success? What is the back-out plan in case of failure?
Asking the right questions will help the organization achieve higher rates of successful changes, which means less unplanned work. And unplanned work is the great weight that’s crushing most low-performing IT organizations. As my friend Jim Hickstein so eloquently put it, “Don’t do: think, type, think, type, think, type, `shit’! Do: think, think, think, type, type, type, `beer’!” Also, coordinating work properly with other business units means less business impact and greater overall availability.
2. Learn lessons: The first part of your change management meetings should be reviewing completed changes from the previous cycle. Pay particular attention to changes that failed or didn’t go smoothly. What happened? How can we make sure it won’t happen next time? What worked really well? Like most processes, change management should be subject to continuous improvement. The only real mistake is making the same mistake twice.
Again the goal of these post-mortems should be to drive down the amount of unplanned work that results from changes in the IT environment. But hopefully you’ll also learn to make changes better and faster, as well as stream-lining the change management process itself.
3. Keep appropriate documentation: Retain all documentation around change requests, approvals, and implementation details. The most obvious reason to do this is to satisfy your auditors. If you do a good job organizing this information as part of your change management process, then supplying your auditors with the information they need really should be as easy as hitting a few buttons and generating a report out of your change management database.
However, where all this documentation really adds value on a day-to-day basis is when you can tie the change management documentation into your problem resolution system. After all, when you’re dealing with an unplanned outage on a system, what’s the first question you should be asking? “What changed?” Well, what if your trouble tickets automatically populated themselves with the most recent set of changes associated with the system(s) that are experiencing problems? Seems like that would reduce your problem resolution times and increase availability, right? Well guess what? It really does.
4. Implement automated controls and demand accountability: If you want people to follow the change management process, they have to know that unplanned changes will be detected and consequences will ensue. As I mentioned above, management is sometimes reluctant to following through on the “consequences” part of the equation. They feel like they’re held hostage to the brilliant IT heroes who are saving the day on a regular basis yet largely ignoring the change management process. What management needs to realize is that it’s these same heroes who are getting them into trouble in the first place. The heroes don’t need to be shown the door, just moved into a role– development perhaps– where they maybe don’t have access to the production systems.
Again, the result is less unplanned work and higher availability. However, it’s also my experience that having automated change controls also teaches you a huge amount about the way your systems and the processes that run on them are functioning. This greater visibility and understanding of your systems leads to a higher rate of successful changes.
The great thing about the steps in Visible Ops is that each step gives back more resources to the organization than it consumes. The first step of implementing proper and useful change management processes is no exception. You probably won’t get it completely right initially, but if you’re committed to continuous improvement and accountability, I think you’ll be amazed at the results.
When benchmarking the high-performing IT organizations identified in Visible Ops, the findings were that these organizations performed 14 times more changes with one quarter the change failure rate of low-performing organizations, and furthermore had one third the amount of unplanned work and 10x faster resolution times when problems did occur. For the InfoSec folks in the audience, these organizations were five times less likely to experience a breach and five times more likely to detect one when it occurred. Further these organizations spent one-third the time on audit prep compared to low-performing organizations and had one quarter the number of repeat audit findings.
If change management is the first step on the road to achieving this kind of success, why wouldn’t you sign up for it?