dev2ops on Twitter
DevOps Toolchain Project
Interested in DevOps?
Search dev2ops

Use DevOps to Turn IT into a Strategic Weapon

As I've worked on the DevOps Cookbook with my co-authors, I've become increasingly conscious of the emphasis of focus of the DevOps community. Lots of attention has been paid to the effects of DevOps within the walls of an IT organization. Far less attention has been paid to the effects of DevOps across the broader company. Almost no attention has been paid the effects of DevOps outside the walls of the company, specifically in relation to other companies and the markets in which you are competing.  

This inward emphasis is understandable considering that DevOps is a movement started by technologists and is a relatively easy sell to other technologists -- breaking down silos, smoothing handoffs, baking in quality, rapid feedback, automating the hell out of everything -- all messages that stand on their own merits.

But if we stop our DevOps evangelism at the boundaries of the IT organization, we are also missing the most valuable return on our DevOps investments. Why? Because the lessons and principles of DevOps can unlock something that is increasingly rare for today’s companies. DevOps can turn your IT Operations into a sustainable competitive advantage for your company. This is the DevOps message needs to be spread throughout the business end of your company, right up to the CEO and the Board of Directors. 

To understand how DevOps can transform IT Operations into a competitive advantage, you first have to look at the current context within which businesses are operating.


Business reality #1: Technology alone cannot provide a competitive advantage

"Technology's potential for differentiating one company from the pack - it's strategic potential - inexorably declines as it becomes accessible and affordable to all"
                                                            Nicholas G. Carr
                                                            From 'IT Doesn't Matter' 


The idea of technology being less and less of a competitive differentiator isn’t a new one. But by its very nature the big shift to online business has only hastened this decline in potential for differentiation.

Your customers are now coming to you via standard interfaces (browser or mobile app store) that connect to you via a standard protocol (http).


So by definition, your business has aligned itself behind a business channel where the point of transaction with your customers is on a level playing field with any of your competitors. 


Business reality #2: Good ideas can and will be copied quickly

Recent high-profile product battles have reinforced the fact that any idea worth copying can and will be copied. Of course, copying ideas has always been a fact of business life. However, a distinct byproduct of our new online world is the increasingly rapid speed at which online ideas can be copied. Having the “better idea” just doesn’t give you much of a lead time. The barriers have become so low that some people are even basing entire business models on this idea (and I’m sure will soon see other people copying that idea as well).



So where can you find competitive advantage?

For the sake of this discussion, let’s put aside branding ("Enjoy Coca-Cola™") and business model strategy (from viral effects to product bundling). These aren’t the focus of this blog and the long term impact for online businesses needs to be discounted because of the effects of business reality #2 described above.

What potential areas for competition are we left with?

1. Scale - removing any limits of data, users, transactions, etc. on the business
2. Lowest Cost - being viewed by your customers as the lowest cost option in the marketplace
3. Sustained Innovation - continuous pace of finding new ways to please customers and improve product market fit

I think the choice might be obvious by now, but let's look at each one of these individually

1. Scale

Scalability, while essential, is not an area in which you can gain a strategic competitive advantage. Scaling of online services is essentially a solved problem. With enough money and enough smart folks (yes, there is a definite art to it) you can scale just about anything. And if you can do it, your competitors can do it too. Cross this off the list.

2. Lowest Cost

Being the low cost provider can look seductively attractive from the perspective of winning an individual sales deal. But when you consider the broader ramifications, life as the low cost provider starts to look a lot more difficult.

First, there can only be one “lowest cost” provider. This automatically sets you up for a high-stakes, winner takes all scenario. Second, hanging your hat on being the lowest cost provider can quickly turn into a race to the bottom, especially in a digital market where the marginal cost of each unit of your product/services is very low. At the end of the day you can’t control how low your competitors will go and ultimately you can’t get lower than free. Lowest cost might be a valid part of your overall business strategy, but it’s limited in its ability to provide a lasting competitive differentiation. Cross this off the list

3. Sustained Innovation

That leaves you with sustained innovation. Unfortunately “sustained innovation” also sounds the most difficult, doesn't it? Consider the “innovation” part first -- do you really have what it takes to be better than your competitors at finding new ways to improve product market fit and delight your customers? Then factor in the “sustained” part -- do you really have what it takes to do that continuously?

But difficult or not, it’s what you are left with. So let’s start breaking it down and figuring out how to turn it into a core competency.


What is innovation?

“Innovation is the specific instrument of entrepreneurship. The act that endows resources with a new capacity to create wealth.”
                                        - Peter Drucker

There are lots of competing ideas on what exactly innovation is or isn’t. Some say the successful application of any new idea counts. Some say it has to be disruptive. Some conflate invention and innovation while others draw a clear distinction between the idea (invention) and innovation (the application).

But not matter which flavor of definition you subscribe to, there is a basic unit of activity supporting innovation. In the context of an online business, that basic unit of innovation activity is taking a business idea and transforming it into a feature that is providing the business feedback in a customer facing environment.

Competition based on innovation all comes down to this unit, or lifecycle. If you can get to a successful result quicker and more reliably than your competitors, a competitive advantage has been achieved.


Innovation is really a numbers game

Of course, everyone wants to believe that the result they will see each time they go through the innovation cycle is unqualified success. But in reality, the outcome is rarely so rosy. We envisage adoring praise from customers and piles of cash, but that rarely materializes.

How bad is it really? The Doblin Group did a global study and found out that only 4% of all innovation efforts meet or exceed targets for return on investment. Booz Allen found “no significant statistical relationships between R&D spending and the primary measures of financial or corporate success”. Even from within “innovation factories” like venture capital portfolios you hear numbers like 80% of companies either exit with minimal returns for investors or fail outright.

So despite our hardwired bias to think otherwise, the vast majority of things we try are just not going to work. From a macro perspective, innovation is a numbers game and the numbers are stacked heavily against us.

To gain an advantage you need to change the numbers in your favor

It’s easy to be mesmerized by the hero’s myth of innovation. This is where the hero musters up everything they can for that one spectacular shot at success. When looking to recreate this story, that one spectacular shot always appears to be a sure thing when it's in mind of a wannabe hero. In reality, that one spectacular shot is really a high stakes gamble that rarely pays out.

Conventional wisdom on how to succeed at innovation is usually along the lines of “think harder and better”. Hire smarter people. Hold more meetings. Get more input. Hold your ideas close to the vest and keep tinkering with it for as long as you can while building up for a big reveal to the marketplace.

Even if you can hire the smartest people in the industry and ensure that you are always working from perfect and timely information, “think harder and better” approach still has you involved in a high stakes game where the odds are still stacked against you. 

Rather than betting it all on those few high stakes gambles, there is a better -- albeit less glamorous --way to win when competing in a numbers game. Give yourself more attempts at finding the right product market fit than your competitors can. Using a sports metaphor, take as many “shots on goal” as you can before the clock runs out.

Keep in mind that these shouldn't be blind shots. You learn a little from each attempt and move quickly to get the next shot off. In the end you’ve tipped the raw odds in your favor and (hopefully) found success before time ran out on the market opportunity (or in a startup’s case, before the cash runs out).


In the example above, we have Company A employing the classic high stakes strategy and Company B playing the increased shots on goal strategy. Company B has gained the ability to safely move through it’s lifecycle 4 times faster than Company A. This gives Company B a distinct advantage. Assuming all other things are equal (same people to hire, same cash to spend, etc.) this means that Company B has 4 times the number of opportunities to find the right product market fit. Increase that number to 8, 16, 32, or more and you can see how this turns into a significant advantage.


The business community already recognizes the opportunity

The idea of finding competitive advantage through continuous innovation is already an established idea in the business community. The Lean Startup movement is built on this concept and makes the case quite eloquently.

"The amount of time a company can count on holding on to market leadership to exploit its earlier innovations is shrinking, and this creates an imperative for even the most entrenched companies to invest in innovation. In fact, I believe a company’s only sustainable path to long-term economic growth is to build an “innovation factory” that uses Lean Startup techniques to create disruptive innovations on a continuous basis."
-Eric Reis
                             The Lean Startup

It’s not just the Lean Startup niche that has gotten onboard with this idea. Across the business landscape you’ll find this message hitting home.

"The world is changing very fast. Big will not beat small anymore. It will be the fast beating the slow.”
                             -Rupert Murdoch
                              Chairman, News Corp

"An organization's ability to learn, and translate that learning into action rapidly, is the ultimate competitive advantage."
                             -Jack Welsh
                              Former CEO, General Electric

"We are already at the stage where almost all of the top managers are very well aware that if they do not lead their company on a process of ongoing improvement it is just a matter of time until you don't have a company. This was not the case 20 years ago. I still remember very much that I had to fight 20 years ago on this concept. Now it's common. Everybody knows it."
                             -Eliyahu M. Goldratt
                              from 'Beyond The Goal' Lecture Series (2005)

While the desire from business leaders to innovate quicker has become clear, what hasn’t been as clear is how IT, now that it's the new "factory floor", will provide the capabilities necessary to enable this rapid learning that leads to rapid innovation.

This is where DevOps comes in.


DevOps is an essential tool for turning IT into a competitive advantage 

At it’s most fundamental level, DevOps is about breaking down silos and removing bottlenecks and risks that screw up an organization's Development to Operations delivery lifecycle. The goal is to enable change to flow quickly and reliably from specification through to running features in a customer-facing environment.

It’s no coincidence that the common illustration for this Development to Operations lifecycle looks a lot like the innovation lifecycle that was previously discussed. By using DevOps ideas to remove the bottlenecks, inefficiencies, and risks from the Development to Operations lifecycle you are directly improving the speed and quality of the innovation lifecycle.


If the mandate of your business is to find competitive advantage through sustained innovation, IT’s mandate must be to provide the business with a fast moving, low friction, high quality lifecycle that enables that innovation advantage. How will you know when you have it moving fast enough? When the Development to Operations lifecycle is no longer the rate limiting step of the company.

Your goal must be to get to the point where you can tell the business “as fast you can tell us what you want and developers can code it, the changes will be live in a customer facing environment giving you direct feedback. And we can can do this with better quality and reliability than you are getting today.”


When you’ve reached that point with your operational capabilities, you will have successfully turned your IT operations into a strategic weapon that gives your company a sustainable advantage over it’s competitors.

One final note... Ignore this call to action at your own peril. Like all competitive advantages, once it has been discovered by the market you can either use it against your competitors or it will be used against you.



Andie Nordgren recaps the CCP Games internal DevOps conference

The following is a guest post by Andie Nordgren, Technical Producer at CCP Games in Reykjavik, Iceland. Andie recently wrote this post as a recap of their internal DevOps conference (for which she was the primary organizer). I thought it was a great overview of what an internal DevOps conference is all about and asked her to share it with our community. -Damon

Service Delivery & DevOps Conference + Hack Sessions

Andie Nordgren
CCP Games 

Our first ever CCP Games Service Delivery & Devops Conference took place last week. It gathered people from many disciplines to talk about how we get our software into customers' hands. Alex Honor and Anthony Shortland of DTO Solutions gave a number of well attended and appreciated talks, and led workshops and hack sessions.



What did we do and how did it go?

Day 1 - Introductions, lectures, and constructive complaining
Day 1 started with introductions over breakfast, and went into the "Setting the Bar" talk, that introduced DTO solutions and current industry trends in releasing software on a fast and lean cycle - and building up a "business immune system" of tests and monitoring that enables that.

The afternoon was spent with a goal setting session, where Andie and Halldor outlined the business reasons that we are looking at how we deliver our service right now - our growing number of services, a much more rapid release cadence, and bigger environments with the introduction of new battleservers and new components.

We talked through capabilities we want to improve as a company, and the opposing forces that hinder us from having those capabilities.

Day 2 - Lectures and tentacles
Day two started with talks from Alex and Anthony about value stream mapping, service delivery, and a model for how to build successful service delivery toolchains.

The afternoon was spend in a "show and tell" where a large group of CCP-ers helped draw up what services are part of our live game, how they are related in terms of versioning and data dependencies, and what tasks they perform. As you can see below the picture had tentacles in it, which is mostly a sign of how central our database is.


Day 3 - Let's do something!
Day 3 started with Anthony and Alex putting 2 stakes in the ground: We would prototype/hack a service delivery tool chain in a HACK sandbox, using Jenkins as a build console, and Rundeck as a deploy console (two key components in the DTO approach for identifying crucial roles in such a toolchain).

We would also attempt to explicitly cover each stage along the top in the Quadrant-diagram from DTO below - develop-commit-build-package-register-promote-install-configure for 1 isolated service in our backend.

After the problem space was talked through, 3 teams were formed to tackle parts of this toolchain:

  • Team Build
  • Team Repo
  • Team Deploy

These teams were various combinations of developers, release managers, build engineers, QA engineers and virtual world sys-admins and leads.

We watched a deployment and kept talking about aspects of our service delivery over lunch. In the afternoon the 3 teams kicked off with their tasks organized from a shared Kanban wall.

A number of standup meetings were held during the day to synch up on progress, and the day ended with many components in place, a refactored build system, a newfound love for Jenkins and all the Jenkins plugins, and a decision for how to integrate Rundeck with our tools using a clever approach that let the tools continue to function in a standalone fashion, and developers write their deployment descriptors in familiar python, but Rundeck could take care of distributed orchestration and having a pretty deploy console web UI.

Day 4 - Hacking, demo and beers
Day 4 was spent continuing the work of getting the full process through for the server, and the day ended with the 3 teams demoing the working components they had each delivered, and how they fit into the Service Delivery Platform, to the wider group of conference participants.

We ended up with server build artifact being created through multiple build steps orchestrated by Jenkins, these artifacts being registered in the Build repository, good progress into using the Deploy commands to spin up a slice and start up the server and CREST as an extended startup-test, before automatically moving the build into a “deployable” area. The Repo served up its contents as options that Rundeck could understand and present to the user, for integrating with the deployment tasks on top of the slice tools, that could now be orchestrated through Rundeck. With a couple of hacky corners cut, as it’s supposed to be in a 1,5 day hack session, but also some solid progress on things we wanted to do anyway, we got a great sense for what a full service delivery platform would look like, and how we can now build towards that kind of environment.

We concluded that some of this work would continue over the summer, and that we will pick it up again in August.

And then of course there was some beer drinking, and all the conversations that didn't fit during the conference itself, as we made a devops invasion of the bar Úrilla Górillan.


What's next?

  • Parts of the work done in the hack sessions will continue and be or is already integrated into MAIN.
  • We will gather up in August after vacations to re-evaluate our tool chain and the stuff we did in the Hack Sessions.
  • We will keep identifying opportunities for continuous improvement, because we now have a language and high level goal in common across teams and disciplines.

Thank you to everyone who participated and to all who continue to carry the service delivery platform and Devops torch in CCP, and a big thank you to Anthony and Alex from DTO for engaging with our tech and teams and sharing their wealth of experience in the form of concrete tools and models.

If you read this far, you've earned one Devops dinosaur.

Over and out
/Andie, Team Rex, Halldor and Thorir B


Integrating DevOps tools into a Service Delivery Platform (VIDEO)

The ecosystem of open source DevOps-friendly tools has experienced explosive growth in the past few years. There are so many great tools out there that finding the right one for a particular use case has become quite easy.

As the old problem of a lack of tooling fades into the distance, the new problem of tool integration is becoming more apprent. Deployment tools, configuration management tools, build tools, repository tools, monitoring tools -- By design, most of the popular modern tools in our space are point solutions.


But DevOps problems are, by definition, fundamentally lifecycle problems. Getting from business idea to running features in a customer facing environment requires coordinating actions, artifacts, and knowledge across a variety of those point solution tools. If you are going to break down the problamatic silos and get through that lifecycle as rapidly and reliably as possible, you will need a way to integrate those point solutions tools. 


The classic solution approach was for a single vendor to sell you a pre-integrated suite of tools. Today, these monolithic solutions have been largely rejected by the DevOps community in favor of a collection open source tools that can be swapped out as requirements change. Unfortunately, this also means that the burden of integration has fallen to the individual users. Even with the scriptable and API-driven nature of these modern open source tools, this isn't a trivial task. Try as the industry might to standardize, every organization has varying requirements and makes varying technology decisions, thus making a once-size-fits-all implementation a practical impossibilty (which is also why the classic monolithic tool approach achieved, on averaged, mixed results at best). 

DTO Solutions has made a name for itself through helping it's clients sort out requirements and build toolchains that integrate open source (and closed source) tools to automate the full Development to Operations lifecycle. Through that work, a series of design patterns and best practices have proven themselves to be useful and repeatable across a variety of sizes and types of companies and environments. These design patterns and best practices have over time become formalized into what DTO calls a "Service Delivery Platform".

I recently sat down with my colleague at DTO Solutions, Anthony Shortland, to have him walk me through the Service Delivery Platform concept.

In this video, Anthony covers:

  • The "quadrant" approach to thinking about the problem
  • The elements of the service delivery platform
  • The roles of various tools in the service delivery platform (with examples)
  • The importance of integrating both infrastructure provisioning and application deployment (especially in Cloud environments)
  • The standardized lifecycle for both infrastructure and applications 

Below the video is a larger version of the generic diagram Anthony is explaining. Below that is an exmaple of a recent implementation of the design (along with the tool and process choices for that specific project).






Using Rundeck and Chef to Build DevOps Toolchains at #ChefConf 2012 (VIDEO)

I presented at #ChefConf 2012 in Burlingame last Thursday on using Rundeck and Chef to Build DevOps toolchains.

The heart of the presentation was a demonstration of continuous build and deployment showing Adam Jacob's chef-rundeck plugin working as a Rundeck resource model source (node provider) and jobs using knife and the Chef server API to manage databag-based application configuration.

At the process level, the presentation connects the dots between service delivery platform design and the loosely-coupled toolchains.

Despite the hotel-wide power outage in the middle of the presentation, the video crew recovered nicely! Below you will find the video and the slides.










High Velocity Release Management with Alex Honor and Betsy Hearnsberger (VIDEO)

This one is for the managers out there who straddle the Dev and Ops divide.

Alex Honor and Betsy Hearnsberger have seen the importance of release management dramatically change over the past decade. Through their collective experiences working inside organizations like E*TRADE,, NASA Ames, and Zynga (as well as Alex's subsequent consulting work at DTO Solutions) they've each amassed a wealth of experience and insight into dealing with high velocity release engineering in large scale and complex organizations.

Since their professional paths have crossed multiple times, I figured I'd get Alex and Betsy together in front of a whiteboard for a chat. In these videos they talk about the common challenges they see, advice for managers addressing these issues, solution approaches that work, and criteria for tool selection.

Please note that these videos were originally shot on July 29, 2011. Due to a technical problem is was thought that these videos were lost. Lucky for us, they have been fully recovered. I'll have to get them both on camera again soon to discuss how their thinking has evolved since then. 


Part 1: Common problems


Part 2: Management's approach to the problems


Part 3: Solution patterns and tool selection