Getty Images
How evolutionary architecture simplified hypothesis driven development
Learn how a new approach to evolving technical architectures promises to make it easier to test out hypothesis about business improvement, application performance, and user experience.
A lot of the buzz around agile development seems focused on how quickly teams can release new code. But a faster pace of release only matters if it makes it easier to test out hypothesis about how changes will affect business goals, application performance, and user experience. An emerging concept called evolutionary architecture makes it easier to make big changes to domains like databases, business processes, service interfaces or user interface without affecting other domains.
"Architecture is abstract until it is operationalized," said Neil Ford, a software architect at ThoughtWorks. Software architects have traditionally thought it was about creating good diagrams for understanding how the pieces fit together. But this misses the fact that cloud infrastructure is improving, better databases keep evolving, and new approaches like microservices and containers show up. "A lot of architects believe that solving architecture is like solving a mathematical equation, but it keeps changing," said Ford.
An evolutionary architecture makes it easier to test out hypothesis at any domain without breaking others. The organization can move forward or roll back a change easily depending on how the experiment goes. These experiments can even involve multiple dimensions.
Ford worked with a mobile car search app in Germany with a mature product. Business analysts kept making suggestions about new features over many years. Ford said, "Every feature made sense but in aggregate it was too complicated for users and as a result the number of users went down. They got into an internal debate about what users wanted they did not know."
Rather than arguing, they set up an experiment in which they built three versions of the app to learn what users wanted. This happens a lot of time in mature products where organizations are not sure if a new feature is making the app better for users or not. Adopting the right set of building blocks for their architecture makes it easier to test out features today and adapt tomorrow if a better combination is discovered.
Use separate domains to reduce coupling
It can be hard to make changes if everything is directly wired to each other, such as the way database connectivity works in a particular codebase. A more organized architecture uses domain layers like presentation, business, services, and databases. The idea is that the enterprise could swap out one layer in a way that limits the impact on others. "We are interested in architectures that let you evolve over multiple dimensions and manage the change," said Ford.
The idea is to make it easy to allow incremental change at a feature level and an architectural level. One good practice is to separate component deployment from feature release using feature toggles, which can be experimented on, monitored, and managed. An incremental change at an architectural level would be to use routing to experiment with releasing new features. When a service is not widely used the route to it can get disintegrated out of the system.
Bake feature toggles into code
Feature toggles make it possible to toggle features on and off. Enterprises are always struggling with setting up new environments for QA that mimic the real environment. Rather than setting up a dedicated QA environment, organizations can use feature toggles so that only testers see the change before it goes out to real users. Feature toggles also make it easier to test out different combination of features side-by-side. This allows an organization to do hypothesis driven development.
There are different types of feature toggles that can be stacked together. These also need to be removed when the experiment is over. Ford recommends organizations create an age of feature toggle to make it easier to trigger developers to remove a feature at the end of an experiment. Organizations need to remove dead code from the code base after it has laid dormant for too long, which could otherwise lead to technical debt. This might seem trivial, but Knight Capital lost hundreds of billions of dollars when dead code was accidentally activated in a new release.
Use fitness functions to guide change
An evolutionary architecture supports incremental guided change as a first principle across multiple dimensions. The core ideas of evolutionary architecture come from evolutionary computing where developers define algorithms that will achieve a result. Fitness functions are used to determine for a given generation if an algorithm is better or worse than the last way they were implemented.
In evolutionary architectures, these algorithms need to do more than see if a new service works better at its intended function. They also need to ensure they don’t introduce new security holes or break governance and risk management requirements. The goal is to set up an architecture that allow developers to change anything they want but provide feedback when the change breaks something else or affects security, scalability, or global performance.
An atomic fitness function can be performed by a unit test that looks at a single characteristic of one deployable artifact to see that it works. Consumer driven contracts are emerging as a common practice in the microservice world for testing integration with other services. These make it easier that changes in one service don’t break the integration with others without having to do constant coordination. This is an example of an integration architecture fitness function that is atomic.
A holistic fitness function makes it easier to see how the architecture behaves as a whole. Netflix’s Chaos Monkey is a great example of a holistic fitness function, which shows what happens if some part of the infrastructure breaks. Ford said, "This is a holistic function because it runs on the general concept of the architecture." When Netflix builds architecture, they need to do so with the awareness that the Chaos Monkey could should down a service at any given time.
Ford said, "The idea is that you are using fitness functions the way people breed dogs. The killer thing in software is the unknown unknowns." An organization might be doing a new service and then Docker comes along with a better way. A good architecture makes it easier to do hypothesis driven development.
Ford explained, "The core of agile methodologies is really about nested feedback loops." One feedback loop that has eluded developers for years is making sense of the user experience. Ford said, "Now with hypothesis and data driven development we can get feedback from users."
