Stone Axes -- the tale of the secret development effort going on at your company right now

UPDATE (7/9/08): Welcome SYS-CON readers!

Talk to any seasoned application developer who's about to embark on a new web application project and you can bet on the following: they have decided on or have narrowed down their choice on an application development framework. The use of some kind of framework is taken for granted. Why do application developers rely on and use frameworks? Because frameworks provide the necessary scaffolding that allows the developer to focus on just the business logic.

Talk to engineers and administrators that manage the online service of a SaaS or eCommerce business, and you see quite a different picture. Indeed, you'll even find a different set of assumptions. This group, the one that keeps the service running, writes and uses custom scripts and tools to get their job done. This "stuff" is software, and it is crucial for keeping the business online. Unfortunately, this software is almost always invisible to the business owners. And, you can also bet on the following: there isn't a framework, and there IS a whole lot of scaffolding being reinvented. When I say framework, I'm not referring to EMS frameworks, the ones that include agents on each host for monitoring and (ahem) "management". What I'm talking about here are the scripts the engineers and administrators write, to automate the online service operations. These are done without an underlying framework. This body of management scripts ultimately boils down to business logic, the logic that governs the delivery of the service operation.

For those that may not be aware of or don't have first hand experience in the SaaS or eCommerce world, it turns out there really are two software development efforts going on. First, there is the software effort everybody knows about - the one the business and product owners focus on - making the software enabling the business model. Second, there is the effort producing all the stuff that enables the delivery of the business, in the form of an always-on online service. This software lets the operations team keep the service updated, maintained and available. Unfortunately, this second body of software is often taken for granted, seems to occur behind the scenes and is almost never subject to the same attention and rigors as the business model software.

The ops folks that work in the trenches, appreciate how important their roles and scripting are to the business. But often times, they don't identify themselves as developers nor do they see their home grown tools as "real" software, but rather just expedient simple time savers. The trouble is this collection of scripts and tools is important to keeping the business running and should be seen as business software. The authors and users of these management scripts appreciate this and do envision better designed, implemented and tested code since their jobs rely on the reliability of their performance. Like any software, requirements change and the software must change with it. What once started off as a simple script, turns into a monolithic, inflexible morass of code.

More often then not, the most used and relied upon programming construct is extremely rudimentary and crude: a loop that iterates over a set of host machines, executing some sequence of system commands. As the scripts importance increases, and the requirements drive its evolution, you also see the need to include logging, notification, security, modularity, data, configuration, etc... Wait. Doesn't this sound like stuff frameworks usually provide?

So what the service operations team needs is a new breed of framework, a domain-specific framework that helps them develop "management applications". Like traditional application development frameworks, it should include features like:

• logging: Log the activity that changes the environment or effects the online service


• notification: Forward events to email or monitoring stations


• security: Control access to only specified actions


• modularity: Allow packaging functionality so it can be reused


• data: Enable the design and maintenance of a data model


• configuration: Allow the management software to be externally configured so it can be used across environments

A domain-specific framework should also include:

• distributed execution: Provide network abstraction so large sets of hosts can be controlled and coordinated


• development tooling: Something like an IDE for development management applications


• packaging and deployment infrastructure: It's a distributed world so the framework should include the ability to package new code modules and deploy them to where they need to go


• canned functionality: There should be a set of common utilities that can be used as a starting point to create new management applications


• multi-language support: certain languages are better suited to different problems so the framework should allow the definition of new components in multiple implementation languages
  

Operations doesn't need to undergo a paradigm shift but rather alignment with their application developer brethren by using a development framework of their own. The business also needs to acknowledge this second, yet equally important software project: the one that keeps the service going.

Comments

[Lee Thompson]

Developers get paid to automate the business process. Not enough attention is paid on automating the technology process which results in the dreaded "throw away code"...

[The IT Skeptic]

It's not just the way we build them, it's also the way we run them. One day Ops will realise that IT staff are IT users and IT tools are production systems, that deserve support, SLAs, continuity, test environments, change control and all the trappings. How can a systems manager console or a service desk not be a production system? but I've seen it often. Informally administered, uncontrolled, unmonitored and unmeasured.

[blah]

I think sometimes the business is afraid of the Sys Admins... They're in high demand and if you bother them, they'll come out of their cave and leave. Nobody doesn't like Sys Admins.

Sys Admins build the systems and attend to them but not many people know what that looks like. If you ask, you're likely to get an incredibly tedious description of duties that will make you frequently look at your watch. That will prove to be in vain because the anti-social creature in charge of your entire collection of machines has no idea that it's your polite way to say, "This is taking a long time and I can't listen any longer."

If you force a "framework" on Sys Admins, it will be ugly. If you let them participate in the process of defining that framework, I think it will be good. But it has to end up better than the shell scripts that have been used for the last 20 years...