Wednesday, February 2, 2011

Technical Debt is a Form of Waste

Technical debt. The term has been popular for a while and is used to refer to a lot of scenarios. It was originally coined by Ward Cunningham to help us recognize that quick and dirty coding sets us up for increased future development effort. It has expanded to include the operational support of code. In my experience, one of the most overlooked forms of technical debt is created when operational support requirements are excluded from the project life cycle.

In many companies, operational support is not considered part of the project life cycle. Operational and project work are often completely decoupled at the lower and middle levels of the IT department, only dovetailing because both divisions report to the CIO or VP of IT. However, this very component of the life cycle, operational support, can deliver the most value to the customer.

This disconnect is created when communication between IT and the rest of the business is limited. When the business defines objectives and timelines for a project with limited or no interaction with the operations team, the operations requirements that are needed to support various projects are presented in an abbreviated form or left out entirely. The project team might not even know that operations requirements exist until the code is almost ready to go live, if then.

I recently participated in an emotionally-charged meeting that was scheduled to review the root cause of the failure of a client-facing system. The project team was angry with the operations team for not supporting their code in a highly-available (HA) environment. The operations team was defensive, stating that they didn’t have the time or resources to address their backlog, including the HA environment.

The project team specified an HA environment in their runbook as part of their handoff to IT operations. The HA system was acknowledged by the operations team as a requirement to adequately support the live code, prioritized it and assigned the creation of an HA system to an operations resource. However, the HA system’s completion was not considered release-gating by the project team, so the project was declared complete as soon as the code was live.

The project team expected their live code to be considered part of the production system, with accompanying up-time SLAs. The IT operations group was working on the HA environment, but didn’t have dedicated resources on the project, so its completion was a long, drawn-out process. The same people working on HA were also doing regular maintenance tasks, working on systems for other projects, and putting out fires.

The key point of failure in this conflict was in not consciously determining the value of the various requirements in the project. It is important in this type of dispute for the business to determine where the customer value delivery exists. Most IT departments don’t consider customer value when negotiating this type of delivery. Without this determination, it is almost impossible to correctly determine where to focus resources.

Part of the miscommunication was created in the beginning of the project, when people from the operations team were not included in the project team. It was exacerbated when the code was permitted to launch without an HA environment. It really snowballed when the project team went on to the next project without fully delivering the value of the code in terms of the delivery life cycle. The operations team, with its own unique requirements, pushed code live without ensuring that the requirements were completed, and then repeated the pattern all over again.

After just a few project lifecycles, it becomes clear that the decoupling of IT operations requirements from business requirements within consecutive projects creates a backlog for IT operations that can’t be surmounted without drastic remediation efforts. Furthermore, this lack of oversight often reveals a lack of understanding of value delivery life cycles. When domains are permitted to hand their delivery over the fence to the next group they tend to value less quality over quantity.

Fortunately, this problem has a simple solution. It’s not easy, because it requires being realistic about how many projects your company can complete in a given amount of time. If you have are in the habit of ignoring IT operations requirements, you might be surprised at how much paying attention to them will slow your project completion rate.

You might have to adjust your understanding of the number of operations people you truly need to support your project work. You need to understand the value you’re creating for your customer is not realized until what you have developed is fully implemented and the customer is finding the value in it. You have done nothing relative to value delivery if you’re not ensuring the product you’re delivering is delivered in full.

In summary, pay attention to three key aspects to prevent a buildup of technical debt as a side-effect of project work:

  1. Make operations requirements release-gating.
  2. Don’t launch on temporary environments to get the product out.
  3. Deliver your product to deliver value, not presence.Bring operations people into the project early and often in the life cycle. In fact, make them an integral part of the life cycle.

Projects are not complete unless the systems on which they live are complete and delivering value to the customer. Anything not providing value to the customer is considered waste. Be the change agent in your organization in halting the creation of technical debt.

Jen Browne and Patrick Phillips

4 comments:

  1. Many organizations are now realizing the need of creating a position that serves as a liaison between the project team and the operations team. The responsibility of that person is to make sure the operations team voice is always present in all planning meetings. They also help the engineers in the project team to take on practices that in the long run will make life easier for the operations team.
    When the project is near completion or a release, the project team engineering will put on their operations engineer hats and start supporting their own product, so they can smooth out the transition to the actual operations team as opposed to just tossing the product over the fence.
    I think this is an approach that works very well given the fact that both the project team and the operations team should work as close as possible to increase communication and obtain great results.
    On the other hand, it is definitively costly to have a group of people who will be playing on both sides of the fence at all times. In the end an organization needs to understand where they want to spend their. If it is up to me I prefer increasing communication and team work and providing more value to the user.

    ReplyDelete
  2. Excellent point, Carlos. Specific ways to address the problem are helpful for others to form their own solutions.

    I particularly like the comment about organizations understanding where to focus. Making it a conscious decision is absolutely essential for increasing value.

    ReplyDelete
  3. While I'm not entirely sure I follow all the detail, I think I get the gist of it. An economist would argue that this was a market failure. Or perhaps more specifically, a failure due to the lack of a market. In essence, think of the project team and the operations team as two separate companies. Then, the solution is actually simplified: the project team needs to contract with the operations team for services. The operations team would, definitionally, then be required to perform a set of services, but they would have some price paid (resources = people etc) to them to deliver this service. The operations team would not (should not) accept a price lower than the actual cost of the resources necessary to accomplish the tasks.

    The main problem here is really this, the project side gets first crack at the resources. They don't always price right (they are under pressure to price low) especially with respect to the operations side. The operations side is operating without contract or specific resources tied to the product, and so strives to minimize costs.

    Economists call the internal market price in a company "transfer pricing."

    This, however, is what happens when markets fail.

    ReplyDelete
  4. Looking at this problem from an economic perspective has really prompted me to shift my thinking. I'm fascinated with the idea of transfer pricing and how it affects my teams. If you have a minute, can you send me suggestions on how to learn more about this? You have my email address. :)

    ReplyDelete

Popular Posts