Taming a Beast

Every day, you struggle with a huge ugly beast that wants to:

• Grow larger
• Get uglier
• Devour you
• Trick you
• Steal your resources
• Waste your time
• Drain your life away 
• Make senseless extra work for you
• Turn what should be easy projects, into difficult projects
• Ruin your reputation with both your customers and your managers
• Prevent you from planning for the future 
• Make your systems unreliable

 

What is this beast?

It is the main disease of complex systems.

Over the years, it has had many names, such as "Bureaucracy" and "Inefficiency".  A currently popular term for it is "Technical Debt". 

In prior posts, we started to explore the causes of this disease, identifying that a primary cause is the intrinsic complexity of the host system.

Complexity

A comprehensive understanding of complexity will take us a long way toward taming our beast.  This is because complexity is at the root of the symptoms we wish to control.

For example, why might it take "too long" to add a new feature to your system? 

• Hard to figure out where to make the change?
• Hard to understand what the system is currently doing?
• Afraid of "breaking" the system?
• Not sure if the change will have unforeseen consequences?

Interestingly, we can prefix each of these with "Because the system is so complex, ":

• Because the system is so complex, its hard to figure out where to make the change
• Because the system is so complex, it is hard to understand what the system is currently doing
• Because the system is so complex, I am afraid of "breaking" the system
• Because the system is so complex, I'm not sure if the change will have unforeseen consequences

In the prior post, we directly related complexity to dependency.  This will turn out to be critical as we develop strategies for coping with complexity.  When applied to the above example, we can develop a good sense for a directly contrasting situation:

• Because there are few dependencies, its easy to figure out where to make the change
• Because there are few dependencies, it is easy to understand what the system is currently doing
• Because there are few dependencies, I am not afraid of "breaking" the system
• Because there are few dependencies, I'm sure the change will have no unforeseen consequences

Methodology

The best overall methodology for combating complexity is to reduce the number of dependencies. The challenge is to accomplish this while minimizing disruption to ongoing operations, and at an acceptable cost.  Luckily, there are effective techniques for doing this.

Next

In the next post, I will discuss a powerful technique for reducing a system's overall complexity by reducing the overall degree of dependency between components.  Ironically, this may sometimes involve increasing the number of dependencies in some areas.

Fuel On The Fire

As I mentioned in the previous post, there are specific factors and conditions that, when present, trigger increasing costs and decreasing efficiency in your systems and software. Addressing these factors is critical.

Specific Factors

What are the factors that cause software to become unmanageable?

Luckily, the same factors are at play in other systems besides software.  This allows us to develop a more comprehensive understanding of the problem, as well as adapt solutions from these other systems.

What other systems show this type of progression toward unmanageability?  Consider these examples:

• Government
• Legal System
• Tax Code
• Large Companies

What do these have in common?  First, they are all large.  Second, they all have a lot of complexity. So, we can conclude that Size and Complexity are major factors.  But, more interestingly, why? Why do these systems have a lot of complexity, and what do we mean by "large"?

Size

What is size?  For our purposes, we will define size as:

Size = The number of entities comprising a system.

Complexity

What is complexity?  The meaning tends to be a bit fuzzy:

• "Hard to understand"
• "Has a lot of parts"
• "Unpredictable"

In order to make headway, we need to do better than this.  Here is a much more useful definition of complexity:

Complexity = The number of dependencies within a system.

This immediately explains why larger systems would tend to be more complex: Larger systems have the potential for more dependencies.

This also explains why older systems tend to be more complex: Older systems have had more time to accumulate both entities and dependencies.

So, it stands to reason: systems that are both old and large would tend to be the most complex - which takes us back to our original list:

• Government
• Legal System
• Tax Code
• Large Companies

All of these are both large and old, host a lot of dependencies - and are therefore complex.

Note: A benefit of these definitions for Size and Complexity is that they are suitable for mathematical treatment and analysis.  Later, we will use this to examine the relationship between complexity and various methodologies for addressing it.

Software

How does this relate to software?

It turns out that software is the perfect environment to host complexity, because:

• It is easy to create entities (get large)
• It is easy to create dependencies between entities
• Because the above is so easy, software ages rapidly, in other words "changes per day" is highly accelerated with software and related systems, effectively "accelerating time" for software-based systems.

Interestingly, complexity is both the "Fuel" and the "Fire", which is why actions taken to fix "out of control systems" often have the opposite effect.

Next

This understanding of complexity points the way toward specific methodologies we can use to control it.

In following posts we will discuss these methodologies and ways to implement them incrementally and in a non-disruptive fashion.

Your Software Owns You

 

You Are Blamed:

• For bugs
• For escalating costs
• For system unreliability
• For not predicting the future
• For attempting to predict the future
• For not planning far enough ahead
• For planning too far ahead
• For taking too much time to make changes
• For making too many changes too quickly
• For not enough testing
• For too much testing
• For not enough configurability
• For too much configurability
• For not enough system flexibility
• For too much system flexibility
• For implementing too many features
• For implementing too few features
• For team sizes being too large
• For team sizes not being large enough
• For difficulty in training new team members

What's Going On Here?

There are many terms for this situation, depending upon the industry or environment being affected.

The bottom line is that when key factors are present, and when conditions are right, this situation is guaranteed to develop.

You may already be familiar with a number of terms associated with this situation:

• Bureaucracy
• Technical Debt
• Committee (effect)
• Over-Engineered
• Calcification
• Mess
• Complex
• Unmanageable
• Legacy
• Intractable
• Fragile
• Unmaintainable
• Costly
• Inefficient
• Confusing

Next

Fortunately, the factors that lead to this situation can be understood, allowing us to apply specific methodologies for both prevention and resolution.

In the next post, we will begin to discuss these factors and their associated methodologies.

Almost Alive

Technology is steadily invading realms that were formerly the exclusive domain of living systems.

As a direct result of steadily increasing complexity, Technology is incrementally acquiring the attributes of life - such as unpredictability (temperament), emergent behaviors (learning), goal-oriented behavior (desire), self-modification (growth), and self-replication (reproduction).

With the advent of software-based systems, Technology has now been released from physical limitations.  This is allowing Technology to move at an increasingly rapid pace. All of us experience this as the exponential acceleration of technologically mediated changes to our daily lives.

This blog is about mastering complex technological systems and software.