Use DevOps to Turn IT into a Strategic Weapon

As I've worked on the DevOps Cookbook with my co-authors, I've become increasingly conscious of the emphasis of focus of the DevOps community. Lots of attention has been paid to the effects of DevOps within the walls of an IT organization. Far less attention has been paid to the effects of DevOps across the broader company. Almost no attention has been paid the effects of DevOps outside the walls of the company, specifically in relation to other companies and the markets in which you are competing.  

This inward emphasis is understandable considering that DevOps is a movement started by technologists and is a relatively easy sell to other technologists -- breaking down silos, smoothing handoffs, baking in quality, rapid feedback, automating the hell out of everything -- all messages that stand on their own merits.

But if we stop our DevOps evangelism at the boundaries of the IT organization, we are also missing the most valuable return on our DevOps investments. Why? Because the lessons and principles of DevOps can unlock something that is increasingly rare for today’s companies. DevOps can turn your IT Operations into a sustainable competitive advantage for your company. This is the DevOps message needs to be spread throughout the business end of your company, right up to the CEO and the Board of Directors. 

To understand how DevOps can transform IT Operations into a competitive advantage, you first have to look at the current context within which businesses are operating.

Business reality #1: Technology alone cannot provide a competitive advantage

"Technology's potential for differentiating one company from the pack - it's strategic potential - inexorably declines as it becomes accessible and affordable to all"
                                                            Nicholas G. Carr
                                                            From 'IT Doesn't Matter' 

The idea of technology being less and less of a competitive differentiator isn’t a new one. But by its very nature the big shift to online business has only hastened this decline in potential for differentiation.

Your customers are now coming to you via standard interfaces (browser or mobile app store) that connect to you via a standard protocol (http).

 

So by definition, your business has aligned itself behind a business channel where the point of transaction with your customers is on a level playing field with any of your competitors. 

Business reality #2: Good ideas can and will be copied quickly

Recent high-profile product battles have reinforced the fact that any idea worth copying can and will be copied. Of course, copying ideas has always been a fact of business life. However, a distinct byproduct of our new online world is the increasingly rapid speed at which online ideas can be copied. Having the “better idea” just doesn’t give you much of a lead time. The barriers have become so low that some people are even basing entire business models on this idea (and I’m sure will soon see other people copying that idea as well).

So where can you find competitive advantage?

For the sake of this discussion, let’s put aside branding ("Enjoy Coca-Cola™") and business model strategy (from viral effects to product bundling). These aren’t the focus of this blog and the long term impact for online businesses needs to be discounted because of the effects of business reality #2 described above.

What potential areas for competition are we left with?

1. Scale - removing any limits of data, users, transactions, etc. on the business
2. Lowest Cost - being viewed by your customers as the lowest cost option in the marketplace
3. Sustained Innovation - continuous pace of finding new ways to please customers and improve product market fit

I think the choice might be obvious by now, but let's look at each one of these individually

1. Scale

Scalability, while essential, is not an area in which you can gain a strategic competitive advantage. Scaling of online services is essentially a solved problem. With enough money and enough smart folks (yes, there is a definite art to it) you can scale just about anything. And if you can do it, your competitors can do it too. Cross this off the list.

2. Lowest Cost

Being the low cost provider can look seductively attractive from the perspective of winning an individual sales deal. But when you consider the broader ramifications, life as the low cost provider starts to look a lot more difficult.

First, there can only be one “lowest cost” provider. This automatically sets you up for a high-stakes, winner takes all scenario. Second, hanging your hat on being the lowest cost provider can quickly turn into a race to the bottom, especially in a digital market where the marginal cost of each unit of your product/services is very low. At the end of the day you can’t control how low your competitors will go and ultimately you can’t get lower than free. Lowest cost might be a valid part of your overall business strategy, but it’s limited in its ability to provide a lasting competitive differentiation. Cross this off the list

3. Sustained Innovation

That leaves you with sustained innovation. Unfortunately “sustained innovation” also sounds the most difficult, doesn't it? Consider the “innovation” part first -- do you really have what it takes to be better than your competitors at finding new ways to improve product market fit and delight your customers? Then factor in the “sustained” part -- do you really have what it takes to do that continuously?

But difficult or not, it’s what you are left with. So let’s start breaking it down and figuring out how to turn it into a core competency.

What is innovation?

“Innovation is the specific instrument of entrepreneurship. The act that endows resources with a new capacity to create wealth.”
                                        - Peter Drucker

There are lots of competing ideas on what exactly innovation is or isn’t. Some say the successful application of any new idea counts. Some say it has to be disruptive. Some conflate invention and innovation while others draw a clear distinction between the idea (invention) and innovation (the application).

But not matter which flavor of definition you subscribe to, there is a basic unit of activity supporting innovation. In the context of an online business, that basic unit of innovation activity is taking a business idea and transforming it into a feature that is providing the business feedback in a customer facing environment.

Competition based on innovation all comes down to this unit, or lifecycle. If you can get to a successful result quicker and more reliably than your competitors, a competitive advantage has been achieved.

Innovation is really a numbers game

Of course, everyone wants to believe that the result they will see each time they go through the innovation cycle is unqualified success. But in reality, the outcome is rarely so rosy. We envisage adoring praise from customers and piles of cash, but that rarely materializes.

How bad is it really? The Doblin Group did a global study and found out that only 4% of all innovation efforts meet or exceed targets for return on investment. Booz Allen found “no significant statistical relationships between R&D spending and the primary measures of financial or corporate success”. Even from within “innovation factories” like venture capital portfolios you hear numbers like 80% of companies either exit with minimal returns for investors or fail outright.

So despite our hardwired bias to think otherwise, the vast majority of things we try are just not going to work. From a macro perspective, innovation is a numbers game and the numbers are stacked heavily against us.

To gain an advantage you need to change the numbers in your favor

It’s easy to be mesmerized by the hero’s myth of innovation. This is where the hero musters up everything they can for that one spectacular shot at success. When looking to recreate this story, that one spectacular shot always appears to be a sure thing when it's in mind of a wannabe hero. In reality, that one spectacular shot is really a high stakes gamble that rarely pays out.

Conventional wisdom on how to succeed at innovation is usually along the lines of “think harder and better”. Hire smarter people. Hold more meetings. Get more input. Hold your ideas close to the vest and keep tinkering with it for as long as you can while building up for a big reveal to the marketplace.

Even if you can hire the smartest people in the industry and ensure that you are always working from perfect and timely information, “think harder and better” approach still has you involved in a high stakes game where the odds are still stacked against you. 

Rather than betting it all on those few high stakes gambles, there is a better -- albeit less glamorous --way to win when competing in a numbers game. Give yourself more attempts at finding the right product market fit than your competitors can. Using a sports metaphor, take as many “shots on goal” as you can before the clock runs out.

Keep in mind that these shouldn't be blind shots. You learn a little from each attempt and move quickly to get the next shot off. In the end you’ve tipped the raw odds in your favor and (hopefully) found success before time ran out on the market opportunity (or in a startup’s case, before the cash runs out).

 

In the example above, we have Company A employing the classic high stakes strategy and Company B playing the increased shots on goal strategy. Company B has gained the ability to safely move through it’s lifecycle 4 times faster than Company A. This gives Company B a distinct advantage. Assuming all other things are equal (same people to hire, same cash to spend, etc.) this means that Company B has 4 times the number of opportunities to find the right product market fit. Increase that number to 8, 16, 32, or more and you can see how this turns into a significant advantage.

The business community already recognizes the opportunity

The idea of finding competitive advantage through continuous innovation is already an established idea in the business community. The Lean Startup movement is built on this concept and makes the case quite eloquently.

"The amount of time a company can count on holding on to market leadership to exploit its earlier innovations is shrinking, and this creates an imperative for even the most entrenched companies to invest in innovation. In fact, I believe a company’s only sustainable path to long-term economic growth is to build an “innovation factory” that uses Lean Startup techniques to create disruptive innovations on a continuous basis."
                            -Eric Reis
                             The Lean Startup

It’s not just the Lean Startup niche that has gotten onboard with this idea. Across the business landscape you’ll find this message hitting home.

"The world is changing very fast. Big will not beat small anymore. It will be the fast beating the slow.”
                             -Rupert Murdoch
                              Chairman, News Corp

"An organization's ability to learn, and translate that learning into action rapidly, is the ultimate competitive advantage."
                             -Jack Welsh
                              Former CEO, General Electric

"We are already at the stage where almost all of the top managers are very well aware that if they do not lead their company on a process of ongoing improvement it is just a matter of time until you don't have a company. This was not the case 20 years ago. I still remember very much that I had to fight 20 years ago on this concept. Now it's common. Everybody knows it."
                             -Eliyahu M. Goldratt
                              from 'Beyond The Goal' Lecture Series (2005)

While the desire from business leaders to innovate quicker has become clear, what hasn’t been as clear is how IT, now that it's the new "factory floor", will provide the capabilities necessary to enable this rapid learning that leads to rapid innovation.

This is where DevOps comes in.

DevOps is an essential tool for turning IT into a competitive advantage 

At it’s most fundamental level, DevOps is about breaking down silos and removing bottlenecks and risks that screw up an organization's Development to Operations delivery lifecycle. The goal is to enable change to flow quickly and reliably from specification through to running features in a customer-facing environment.

It’s no coincidence that the common illustration for this Development to Operations lifecycle looks a lot like the innovation lifecycle that was previously discussed. By using DevOps ideas to remove the bottlenecks, inefficiencies, and risks from the Development to Operations lifecycle you are directly improving the speed and quality of the innovation lifecycle.

If the mandate of your business is to find competitive advantage through sustained innovation, IT’s mandate must be to provide the business with a fast moving, low friction, high quality lifecycle that enables that innovation advantage. How will you know when you have it moving fast enough? When the Development to Operations lifecycle is no longer the rate limiting step of the company.

Your goal must be to get to the point where you can tell the business “as fast you can tell us what you want and developers can code it, the changes will be live in a customer facing environment giving you direct feedback. And we can can do this with better quality and reliability than you are getting today.”

When you’ve reached that point with your operational capabilities, you will have successfully turned your IT operations into a strategic weapon that gives your company a sustainable advantage over it’s competitors.

One final note... Ignore this call to action at your own peril. Like all competitive advantages, once it has been discovered by the market you can either use it against your competitors or it will be used against you.

High Velocity Release Management with Alex Honor and Betsy Hearnsberger (VIDEO)

This one is for the managers out there who straddle the Dev and Ops divide.

Alex Honor and Betsy Hearnsberger have seen the importance of release management dramatically change over the past decade. Through their collective experiences working inside organizations like E*TRADE, Ask.com, NASA Ames, and Zynga (as well as Alex's subsequent consulting work at DTO Solutions) they've each amassed a wealth of experience and insight into dealing with high velocity release engineering in large scale and complex organizations.

Since their professional paths have crossed multiple times, I figured I'd get Alex and Betsy together in front of a whiteboard for a chat. In these videos they talk about the common challenges they see, advice for managers addressing these issues, solution approaches that work, and criteria for tool selection.

Please note that these videos were originally shot on July 29, 2011. Due to a technical problem is was thought that these videos were lost. Lucky for us, they have been fully recovered. I'll have to get them both on camera again soon to discuss how their thinking has evolved since then. 

Part 1: Common problems

Part 2: Management's approach to the problems

Part 3: Solution patterns and tool selection

DevOps Lessons from Lean: Small Batches Improve Flow

DevOps problems are fundamentally flow problems. Work doesn't flow properly from one end of the lifecycle (Dev) to the other end of the lifecycle (Ops).

While spirited discussions on tools are a regular occurrence in DevOps circles, there are other simple, yet profound, techniques that have nothing to do with technology but have proven to have a huge impact on improving flow.

Top of that list? Work in small batches.

It seems so simple that it couldn't possibly make that big of difference, but it does. And there is historical precedent for it as well. The principle of working is small batches has proved it's merit in Agile software development and on an even larger stage during the manufacturing revolutions of the 1970s and 1980s.

The reasons why working in small batches has such a strong net positive impact on flow might seem a bit counterintuitive at first. In the absence of relying on "because I told you so", below are the best explanations I could find as to why this works.

What is a "batch size"?
A batch is the unit of work that passes from one stage to the next stage in a process. The batch size the scale of that work product.

What are the benefits of reducing batch sizes?

Reduces cycle time and gets you quicker feedback - With a small batch size, each batch makes it through the full lifecycle quicker. Since work on a feature isn't complete until it is successfully running in production and getting feedback from users, large batch sizes simply delay that feedback. This means the larger the batch the longer you wait to find out if you did it right. It's easier to make business and technical decisions and easier to recover from a mistake if you are working on shorter time horizons.

Reduces risk of an error or outage - With a small batch size, you are reducing the amount of complexity that has to be dealt with at any one time by the people working on the batch. The reduction in complexity comes not only from the number and size of the moving parts that are touched while working on the batch, but also in the amount of person-to-person communication that needs to happen (due to smaller teams). This is just acknowledging the natural limitations of human beings. The more complexity people have to deal with, the more mistakes there will be. Smaller batch size also leads to quicker feedback, so if there is an error in the batch it will be caught sooner. A small batch size lends itself well to quicker problem detection and resolution (the field of focus in addressing the problem can be contained to the footprint of that small batch and the work that is still fresh in everyone's mind).

Reduces product risk - This builds on the idea of faster feedback. The sooner you can put an individual feature in front of your target audience, the sooner you will know if you've achieved the right product and market fit. The larger the batch size, the greater the product risk when you finally release that batch. Statistics shows us that it's beneficial to decompose a large risk into a series of small risks. For example, bet all of your money on a single coin flip and you have a 50% chance of losing all of your money. Break that bet into 4 smaller bets and it would take 4 sequential bets to result in financial ruin (1 in 16 or 6.25% chance of losing all of your money).

Large batch sizes also often lead to compounding schedule delays and cost overruns. The larger the batch, the more likely it is that a mistake was made in estimating or during the work itself. The chance and potential impact of these mistakes compounds as the batch size grows… increasing the delay in being able to get that all important feedback from the users and increasing your product risk.

Improves efficiency and lowers overhead - Conventional wisdom holds that large batches allow greater productivity (i.e. you get more done with large uninterrupted periods of work) and lower overhead (less batches = less transactional costs). As has been proven in the manufacturing world (Lean) and now software development (Agile), this simply isn't the case. The larger the scope of the batch, the more complexity the individual has to deal with. The complexity of a debug task grows as 2ⁿ when n things are changed in one batch. In knowledge work, the larger the uninterrupted period of work leads to greater change complexity, greater the volume of debug work, and more handoff complexity. That is all added overhead. But even assuming the individual was still being more efficient by working in a large batch, you would still be creating greater inefficiency for the end-to-end process.

For a large batch of changes, especially those made to an even larger system, the handoff to the next step in the process is going to be highly inefficient for the receiving party to deal with (think: Development to Operations "toss if over the wall" handoff of a major release). And if something goes wrong, the time between when the error was introduced and when it will be discovered is so long that it is no longer fresh in the mind of the person who introduced the error. Small batches also have been proven to actually reduce transaction costs because of a curious fact of human nature… people get better at and find ways to increasingly improve the things they are forced to do more often.

Improves management visibility and control - Reducing batch sizes gives you a greater number of instrumentation points by which you can visualize and measure the flow of work through your organization. It's notoriously difficult to accurately determine progress of in-flight work. You are largely going to be limited to the subjective analysis of project managers and the biased opinion of the person doing the work. The only points where you can have certainty is either when the work has just started or when the work has just completed (and accepted by the next step in the process). With large batch sizes you have to wait long periods of time between those start and completion points, making it difficult to see how things are flowing, providing little guarantee that you will have adequate warning if things are going wrong, and allowing for few opportunities to make adjustments to optimize or triage. With small batch sizes you can see work move through the lifecycle with certainty, spot problems early, and make ongoing adjustments to optimize the flow of delivery.

Encourages decoupled architectures with less dependency issues - Smaller batch sizes can also have a positive impact on architecture. Most IT systems are built from within the context of large projects. Large projects create them and then large projects are undertaken to change them. The result is a built-in tolerance for monolithic architectures with complex dependencies. As you move to small batch sizes you are naturally limiting the work in progress on a particular segment of your code/infrastructure. While initially this might seem like it will slow the organization down, the principles of flow show that this will actually give you greater throughput over time. But in order to speed things up even further, you will end up looking for ways to increasingly decouple and isolate (including making fault tolerant) your architecture to allow for greater parallelization of work.

What are the economic benefits of reducing batch size?
In manufacturing and in software development, reducing batch sizes has been showen to have a significant impact on the economics of the production process. The diagram below (scanned from Donald G. Reinertsen's "The Principles of Product Development Flow", pg 121) lays out the direct links between smaller batch sizes and improved economics. I think the logic speaks for itself.

What are your control points for reducing batch sizes?
Reducing batch sizes is a policy decision that needs to be implemented at multiple levels: 

Project Initiation and Funding - How projects are formed and funded tends to have a strong correlation to batch size. The definition of requirements and success criteria, in addition to the allocation of budget, is usually done in a large batch that corresponds to a specific or set of business goals that were created at the quarterly or yearly scale. The inertia of this large batch is often carried throughout the rest of the lifecycle, becoming a pacemaker of sorts that encourages large batch sizes. Positive work done to break down these large initial batches into smaller batches can turn that inertia back into a net positive effect for the company. Reduction in the time horizon for the expected results of a project is usually a good way to force the issue (e.g. try scoping and budgeting projects to single month size rather than quarter/multi-quarter size).

Project management - When creating projects consider what is the smallest amount of change that can be undertaken in the shortest amount of time and still achieve a measurable result. This will naturally lead to smaller teams working on smaller batches of work that can flow independently through the lifecycle with faster feedback and lower risk to the overall system.

Testing - Demand that individual pieces of work are tested as soon as those pieces of work are completed (and not wait for the entire project/release to be code complete). Continuous integration and it's built in unit/smoke tests is a crude example of this principle. Carry that further. Ensure that full deployment and testing efforts are ongoing during any project. This will automatically force engineers to think about their work in small units that can be completed and handed off for testing at regular intervals (naturally creating the urge to reduce batch sizes).

Release management - Break down large releases into small units of deployment that employ standardized packaging and configuration management mechanisms. These units of deployment should be aligned towards the things that are changed (i.e. application services) rather than large project releases that change many things. In addition to reducing deployment and configuration woes, this also has the effect of standardizing batch sizing across lifecycle by determining the appropriate unit of change for your infrastructure.

I'm standing the on shoulders of people a lot smarter than me in this post. If you are interested in these ideas please check out:
http://www.amazon.com/Principles-Product-Development-Flow-Generation/dp/1935401009/ref=cm_cr_pr_product_top
http://www.startuplessonslearned.com/2009/02/work-in-small-batches.html http://www.dbrmfg.co.nz/Production%20Batch%20Issues.htm
http://www.informit.com/articles/article.aspx?p=1833567&seqNum=3

Value of DevOps Culture: It's not just hugs and kumbaya

The importance of culture is a recurring theme in most DevOps discussions. It's often cited as the thing your should start with and the thing you should worry about the most.

But other than the rather obvious idea that it's beneficial for any company to have a culture of trust, communication, and collaboration... can using DevOps thinking to change your culture actually provide a distinct business advantage?

Let's take the example of Continuous Deployment (or it's sibling, Continuous Delivery). This is an operating model that embodies a lot of the ideals that you'll hear about in DevOps circles and is impossible to properly implement if your org suffers from DevOps problems. 

Continuous Deployment is not just a model where companies can release services quicker and more reliably (if you don't understand why that is NOT a paradox, please go read more about Continuous Deployment). Whether or not you think it could work for your organization, Continuous Deployment is a model that has been proven to unleash the creative and inventive potential of other organizations. Because of this, Continuous Deployment is a good proxy for examining the effects of solving DevOps problems.

Eric Ries sums it up better than I can when he describes the transformative effect that takes place the further you can reduce the cost, friction, and time between releases (i.e. tests to see if you can better please the customer).

"When you have only one test, you don’t have entrepreneurs, you have politicians, because you have to sell. Out of a hundred good ideas, you’ve got to sell your idea. So you build up a society of politicians and salespeople. When you have five hundred tests you’re running, then everybody’s ideas can run. And then you create entrepreneurs who run and learn and can retest and relearn as opposed to a society of politicians."

-Eric Ries
  The Lean Startup (pg. 33) 

 
That's a business advantage. That's value derived from a DevOps-style change in culture.

Huddle.com - Being Agile about Agile with Andy McLoughlin

Last week I attended a Pacific Crest Mosiac Summit in Vail Colorado and met with a number of institutional technology investors.  During the summit there were a number of interesting BOF’s and I was fortunate enough to moderated a #devops BOF.  Andy McLoughlin the founder of Huddle.com, was one of the individuals in the Devops BOF and he blew me away with some of his internal #devop practices.  I am happy to share some of Andy’s insights here with you in the video.