Shine the Light

If you want to know what's really going on, shine a light on it...

As I mentioned in the prior post, there's a powerful approach for reducing complexity that allows a significant number of objects to be removed from your system, while decreasing maintenance and subsequently improving time-to-market for new features and functionality.

Sounds almost too good to be true!  Fortunately, not only is it true, but it is likely that you already leverage this technique in a limited fashion.

The Way It Used To Be

To fully describe this technique, I would first like to take you back in time.  Back to the misty, barely remembered shadows of antiquity.  Back to the dawn of software as we know it, maybe five years ago.

Back then, we were still connecting our systems to one-another using outdated techniques like proprietary file formats, serialized classes, fixed-length fields, proprietary web services, custom objects, specialized record formats, etc.

Back then, we didn't understand that this created tight dependencies between our systems.  For example, if a proprietary file format needed to be changed, every system consuming that file format would also need to change - and change in a tightly synchronized fashion, or you would end up supporting multiple legacy formats.

Back then, we didn't understand that the dependencies between our systems were holding us back. That we were all afraid of changing our system interfaces and triggering the inevitable firestorm of flaming dominoes, with each one falling and igniting the next in a roaring inferno of expensive and disruptive change.

The Way It Is Now

We have learned a lot since then. Now, we know better.  We connect our systems to one another using universal and standardized formats capable of accommodating any data structure.  Today, we know how to use JSON and XML to connect our systems in a clean way that readily accommodates change.

Managing and communicating lengthy and complex inter-system communication and object standards are a thing of the past.  It has all been reduced to a simple schema that is easily shared, and even can be largely ignored - except for just the pieces that you are interested in, or that are absolutely necessary.

In a matter of minutes, you can now spin up a REST interface and have the entire world happily making calls into your system with minimal documentation or fuss.

So What?

Why is this important?  First, its not completely true:

Many companies continue to use legacy methods of connecting their systems, both externally and internally.

And here's the big one:

It's all a hierarchy. It is literally objects all the way down.

For all of the same reasons that virtually everyone is now using REST with JSON to implement system interfaces, the same benefits can be realized FOR ANY SOFTWARE OBJECT, including internal components, packages, namespaces, classes, files, configuration, tables, records, fields, properties, databases, etc.

And the icing on the cake:

It's easy to implement.  In fact, it is easier to implement that what we have been doing.

Transparency

Proper implementation of this technique first involves a more complete understanding of what we are actually doing when we implement a JSON REST service, which is just an implementation of Transparency.

Transparency is:

The ability of an object to allow its inputs to be passed through to its outputs – even when the inputs change unpredictably.

"It's gotta be transparent to let the light through!" 

Transparency allows objects to be "transparent" to data.  This is extremely desirable in multiple ways, allowing the object:

• To be maximally immune to changes in the data
• To ignore data that it is not concerned with
• To be backward compatible with new data
• To implement forward compatibility with unimplemented data changes
• To add information to the data without repercussion
• Reduce or eliminate interfaces that it needs to "know about"

As a result of these benefits:

• Many custom objects in a standard software stack can be eliminated (in exactly the same way that we can eliminate a custom file-format - and most of its associated supporting custom software - by using JSON or XML).
• Changes in one part of the system have less impact on the rest of the system
• Changes to data structure may require little or no software changes.  For example, adding a column to a table would make its data immediately available in a view - without any software changes. 
• Dependencies are reduced
• Maintenance is reduced
• System reliability is improved
• Time-to market is improved

Implementation of Transparency

There are a number of ways to implement transparency in any system.  In all cases, we gain the benefits of decoupled components, improved extensibility, better accommodation of change, and elimination of custom data formats, custom objects and/or custom interfaces.

Transparency in interfaces is being adopted rapidly.  It is likely that you are familiar with one or more of the following implementations:

• Exposing interfaces (Such as an external REST/JSON interface.)
• Internal system interfaces (Such as a microservice architecture implemented as REST/JSON)
• Configuration (Such as an XML configuration file.)
• Component interfaces Various open-source or commercial component interfaces exist, such as JCR, JMQ, Database interfaces, Message queues, etc. that use JSON or XML as the payload)
• As a field (such as as a JSON column in a database)

The Universal DTO

One of the most effective ways of implementing transparency is by replacing DTO's (data transfer objects) with a universal DTO (such as some form of JSON object).

Custom DTO's represent a particularly insidious form of technical debt and dependency.

This is because they were once absolutely essential for bridging the gap between proprietary interfaces and your own internal environment, and preventing a complicated proprietary interface from "poisoning" your pristine internal systems. Because of this, we were all taught to make vast numbers of them.

Sadly, it is even common practice for DTO's to fully surround a transparent interface (such as a REST/JSON interface) and thereby nullify much of the benefit of using JSON in the first place.

Consider the following standard client/server stack:



Notice that data is being supplied to, and consumed from, the REST API using DTO's.  This means that any changes to the data structure will also require changes to these two DTO's.  This situation can be easily avoided, and the system simplified, by replacing the custom DTO's with a universal DTO, such as a JSON Object.

After all, data being supplied by the server needs to be converted from the DTO to JSON anyway. Likewise, on the client side, the data needs to be converted yet again into a DTO.

In their purest form, DTO's represent just data - especially if you have done a good job of keeping them separate from your entities (models).  As pure data, they are virtually name/value pairs, becoming little more than technical debt (in the form of maintenance), and are a major impediment to implementing transparency.

A Better Way

A simpler (and transparent) approach is to simply create the JSON object on the server, transmit it to the client, which then passes it on to the business logic and the view.  No need to convert it twice - both to, and from, custom classes.  In this way, these custom DTO's can be eliminated, while at the same time improving the transparency, simplicity, and speed of the system:




But we shouldn't stop there.  The business logic on each side can pass the Universal DTO through. Allowing other objects unimpeded access to the data:




Now the client view is finally receiving unobstructed access to the data.  On the server, the other DTO is no longer needed, since the universal DTO fulfills the same function.  This further simplifies and improves our system:




Also, the result of a query (a data set) is already delivered from the database in a universal form. (As long as we don't prematurely trap our data in objects via a high-maintenance framework like Hibernate or Entity Framework).  It is a simple exercise to convert the result-set to JSON (in fact,  many databases will already do this for us):

We have now achieved a transparent stack. Not only is it simpler, more efficient, easier to understand, and easier to maintain.  It also fully implements transparency.  This means that a column added to the table is immediately available to be used in the view - with no code changes needed in between.

A Light at the End of the Tunnel

Transparency is a powerful tool for reducing the complexity of your system by reducing both dependencies and entities, while at the same time improving performance and making future change easier to implement.

The best implementations of transparency:

• Implement a universal DTO object (UDTO)
• Implement the UDTO as not just a dictionary or a map, but rather as an encapsulation of JSON or other hierarchical collection.
• Use the UDTO for many major internal and external decoupled interfaces
• Allow many objects to be instantiated using a UDTO
• Manage configuration using a UDTO

Next Time

In my next post, we'll explore a number of techniques for extending the management of complexity and dependencies to teams and processes.

Engine Repair at 60 MPH

You can't stop the car to do repairs, so... 

As I mentioned in prior posts, complex systems are characterized by their dependencies - both internal and external.

In practical terms, this means that changes to one part of a complex system have the potential for unforeseen consequences in other parts of the system.

This leads to two common approaches to system maintenance.  Sadly, both of these approaches ultimately create unacceptable problems:

1.   Additive Change

In this approach, developers and managers continuously make the decision that it is safer to "patch", "tweak", or "adjust" a critical part of the system, rather than refactor or replace it.

This "safe" approach involves adding yet another layer to the system or component, and generally creates more complexity and more dependencies. This reduces the maintainability of the system, while increasing technical debt.

Ironically, "patching" tends to be applied to the most actively-changing parts of the system.  This results in the greatest accumulation of technical-debt in precisely the areas that need greatest maintenance velocity!

2.  Monolithic Development and Deployment

In this approach, developers and managers decide that the only "safe" approach is to treat the entire system as a single unit (since there are so many dependencies).

This perspective leads to laziness with respect to OO (object oriented) design and practices, and is characterized by the following symptoms:

• Insufficient separation of concerns

• No real appetite for componentization

• Lack of well defined APIs both within and between components

• Reduced independence of teams (teams are encouraged to know more about one-another's work, rather than less)

• Massive technical meetings (involving more than six people)

• Favoring system-wide, end-to-end regression testing, rather than targeted unit testing

• Too many conflicting and confusing configuration settings/parameters

• Monolithic and fragile deployment, involving most or all of the system at once

• A general belief that separating components via API's and interfaces is "extra work" that is not realistic or warranted.

This approach severely reduces the velocity of development and deployment, while increasing technical debt.

A Better Approach

In order to reduce complexity, it is critical to eliminate dependencies where possible.  This is the key to creating components that maximize maintainability, minimize team size, maximize independence from other components - including an independent life cycle - ultimately resulting in parallel development, maintenance and deployment.

However, an important consideration is that we need to do this while reducing or eliminating disruption to ongoing operations.

In other words, we want to do engine repair on our car while racing down the highway at 60 mph!

There is an incremental step-by-step way to accomplish this (without crashing).

By The Numbers

The following steps allow us to reduce the complexity of our systems, in a controlled and safe way:


1.  Discover

Identify parts of the system that have the potential for independence.  Examples include Database, User Interfaces (front-end), Business Logic, External Interfaces, Analytics, Transaction Processing, Data Transformation, etc.

Currently, there may be large numbers of dependencies between these parts of your system.  Our goal in this step is just to identify the ideal components (concerns).  We will address the dependencies later.

 
2.  Decide

With the key components of the system identified, a decision is made for each component: whether or not to create a new (virtually empty) component, or use the existing component.  This decision depends upon how much technical debt a given existing component has.  Components that are extremely debt-ridden should not be (flagged to be) kept.

The key point here is that highly debt-ridden components cannot easily be refactored without risk to ongoing operations.  In these cases, it is better to make a new component that will exist (for a time) in parallel with the old component.  These new components should start life virtually empty - not being used by the system (even after initial deployment to production).


3.  Connect

At this point, all components will have been identified.  Some of these components will already be in use by the system (because they existed previously), and some components will be new - and not yet being used.

Next, design and implement clean interfaces between your components.  These will become the "contracts" used by developers going forward, allowing components (and their associated teams) to work independently and in parallel (to the interface).

A key point here is that these interfaces should be minimal - as thin as you can possibly make them.

Any newly implemented interfaces will not need to be used until later. 

4. Change

With the components and interfaces identified and/or implemented, there will now be a framework for incremental, and non-disruptive change.

The main point is that there should now be well-understood interfaces and components that are ready to receive legacy code and iterative changes that will reduce dependencies and the resulting complexity and technical debt.

Next, incrementally make the following types of changes:

• Remove unnecessary dependencies between components (by moving legacy code/data, and/or by taking advantage of any new interfaces and components.)

• Add functionality to new components (by mining legacy functionality from other parts of the system and/or adding new or newly refactored functionality)

• Remove unnecessary copies of existing code/functionality (by redirecting dependencies to newly identified and implemented interfaces and components)

An important point is that these steps can be done incrementally, and deployed to the production environment - even if the new code or new functionality is not being used.  This is important in order to avoid massive testing and massive switch-overs.  

Later, relatively small subsets of system functionality can be directed to the incrementally forming new infrastructure.  This allows confidence in the new components and structure to grow gradually and in a controlled, non-disruptive fashion.

Another important point is that this process allows teams and systems to gradually become adapted to a more incremental and parallel mode of development and deployment.

5. Use

With a growing new infrastructure (existing in parallel with the legacy structures), it will soon become possible to redirect a subset of functionality to new or improved components.

This may take the form of a single customer/client (who may have requested new functionality), or it may take the form of a subset of system functionality, such as backup, transaction processing, table maintenance, an external interface, etc.

The key point is that system functionality can be switched over gradually, in a controlled manner, and with the ability to immediately switch back if there are any issues.

This mitigates any risk associated with change, opening the door to more confidence in accommodating change, and allowing teams and managers to become less conservative about change - and ultimately making "patching" largely a thing of the past - only used as a last resort - and only temporarily.

6. Retire

Over time, parts of the existing (legacy) structures and functionality will become unused.  These parts of the system may be left in place as long as is needed to cement confidence in new components and functionality.

Eventually, these (debt-ridden and now obsolete) parts of the system can be retired (removed).

Next Time

In my next post, I will present another powerful approach for reducing complexity that typically allows a significant number of objects to be removed from the system, while decreasing maintenance and subsequently improving time-to-market for new features and functionality.

Taming a Beast

Every day, you struggle with a huge ugly beast that wants to:

• Grow larger
• Get uglier
• Devour you
• Trick you
• Steal your resources
• Waste your time
• Drain your life away 
• Make senseless extra work for you
• Turn what should be easy projects, into difficult projects
• Ruin your reputation with both your customers and your managers
• Prevent you from planning for the future 
• Make your systems unreliable

 

What is this beast?

It is the main disease of complex systems.

Over the years, it has had many names, such as "Bureaucracy" and "Inefficiency".  A currently popular term for it is "Technical Debt". 

In prior posts, we started to explore the causes of this disease, identifying that a primary cause is the intrinsic complexity of the host system.

Complexity

A comprehensive understanding of complexity will take us a long way toward taming our beast.  This is because complexity is at the root of the symptoms we wish to control.

For example, why might it take "too long" to add a new feature to your system? 

• Hard to figure out where to make the change?
• Hard to understand what the system is currently doing?
• Afraid of "breaking" the system?
• Not sure if the change will have unforeseen consequences?

Interestingly, we can prefix each of these with "Because the system is so complex, ":

• Because the system is so complex, its hard to figure out where to make the change
• Because the system is so complex, it is hard to understand what the system is currently doing
• Because the system is so complex, I am afraid of "breaking" the system
• Because the system is so complex, I'm not sure if the change will have unforeseen consequences

In the prior post, we directly related complexity to dependency.  This will turn out to be critical as we develop strategies for coping with complexity.  When applied to the above example, we can develop a good sense for a directly contrasting situation:

• Because there are few dependencies, its easy to figure out where to make the change
• Because there are few dependencies, it is easy to understand what the system is currently doing
• Because there are few dependencies, I am not afraid of "breaking" the system
• Because there are few dependencies, I'm sure the change will have no unforeseen consequences

Methodology

The best overall methodology for combating complexity is to reduce the number of dependencies. The challenge is to accomplish this while minimizing disruption to ongoing operations, and at an acceptable cost.  Luckily, there are effective techniques for doing this.

Next

In the next post, I will discuss a powerful technique for reducing a system's overall complexity by reducing the overall degree of dependency between components.  Ironically, this may sometimes involve increasing the number of dependencies in some areas.

Fuel On The Fire

As I mentioned in the previous post, there are specific factors and conditions that, when present, trigger increasing costs and decreasing efficiency in your systems and software. Addressing these factors is critical.

Specific Factors

What are the factors that cause software to become unmanageable?

Luckily, the same factors are at play in other systems besides software.  This allows us to develop a more comprehensive understanding of the problem, as well as adapt solutions from these other systems.

What other systems show this type of progression toward unmanageability?  Consider these examples:

• Government
• Legal System
• Tax Code
• Large Companies

What do these have in common?  First, they are all large.  Second, they all have a lot of complexity. So, we can conclude that Size and Complexity are major factors.  But, more interestingly, why? Why do these systems have a lot of complexity, and what do we mean by "large"?

Size

What is size?  For our purposes, we will define size as:

Size = The number of entities comprising a system.

Complexity

What is complexity?  The meaning tends to be a bit fuzzy:

• "Hard to understand"
• "Has a lot of parts"
• "Unpredictable"

In order to make headway, we need to do better than this.  Here is a much more useful definition of complexity:

Complexity = The number of dependencies within a system.

This immediately explains why larger systems would tend to be more complex: Larger systems have the potential for more dependencies.

This also explains why older systems tend to be more complex: Older systems have had more time to accumulate both entities and dependencies.

So, it stands to reason: systems that are both old and large would tend to be the most complex - which takes us back to our original list:

• Government
• Legal System
• Tax Code
• Large Companies

All of these are both large and old, host a lot of dependencies - and are therefore complex.

Note: A benefit of these definitions for Size and Complexity is that they are suitable for mathematical treatment and analysis.  Later, we will use this to examine the relationship between complexity and various methodologies for addressing it.

Software

How does this relate to software?

It turns out that software is the perfect environment to host complexity, because:

• It is easy to create entities (get large)
• It is easy to create dependencies between entities
• Because the above is so easy, software ages rapidly, in other words "changes per day" is highly accelerated with software and related systems, effectively "accelerating time" for software-based systems.

Interestingly, complexity is both the "Fuel" and the "Fire", which is why actions taken to fix "out of control systems" often have the opposite effect.

Next

This understanding of complexity points the way toward specific methodologies we can use to control it.

In following posts we will discuss these methodologies and ways to implement them incrementally and in a non-disruptive fashion.

Your Software Owns You

 

You Are Blamed:

• For bugs
• For escalating costs
• For system unreliability
• For not predicting the future
• For attempting to predict the future
• For not planning far enough ahead
• For planning too far ahead
• For taking too much time to make changes
• For making too many changes too quickly
• For not enough testing
• For too much testing
• For not enough configurability
• For too much configurability
• For not enough system flexibility
• For too much system flexibility
• For implementing too many features
• For implementing too few features
• For team sizes being too large
• For team sizes not being large enough
• For difficulty in training new team members

What's Going On Here?

There are many terms for this situation, depending upon the industry or environment being affected.

The bottom line is that when key factors are present, and when conditions are right, this situation is guaranteed to develop.

You may already be familiar with a number of terms associated with this situation:

• Bureaucracy
• Technical Debt
• Committee (effect)
• Over-Engineered
• Calcification
• Mess
• Complex
• Unmanageable
• Legacy
• Intractable
• Fragile
• Unmaintainable
• Costly
• Inefficient
• Confusing

Next

Fortunately, the factors that lead to this situation can be understood, allowing us to apply specific methodologies for both prevention and resolution.

In the next post, we will begin to discuss these factors and their associated methodologies.

Almost Alive

Technology is steadily invading realms that were formerly the exclusive domain of living systems.

As a direct result of steadily increasing complexity, Technology is incrementally acquiring the attributes of life - such as unpredictability (temperament), emergent behaviors (learning), goal-oriented behavior (desire), self-modification (growth), and self-replication (reproduction).

With the advent of software-based systems, Technology has now been released from physical limitations.  This is allowing Technology to move at an increasingly rapid pace. All of us experience this as the exponential acceleration of technologically mediated changes to our daily lives.

This blog is about mastering complex technological systems and software.