Effective DevOps pipelines require a continuous delivery tool chain

Effective DevOps and pipeline identification

The pathway to effective DevOps starts by identifying the various pipelines you have to work with.

The exact type, number and nature of all of your pipelines will vary from organization to organization. A typical company might have the following pipelines feeding its software delivery process:

• A software architecting pipeline that designs new features.

• A coding pipeline, which handles the implementation of new features.

• A quality assurance pipeline, where new features are tested for technical stability.

• A user experience pipeline, where changes are tested for usability.

• A staging and build pipeline, in which new code is configured and compiled.

• A deployment pipeline that pushes new builds into production.

While some of these pipelines involves processes that take place earlier in the software delivery lifecycle than others, the fact is that you can’t neatly line them up in chronological order. Design might come before coding, but different kinds of software testing occur at the same time. Staging and building may also happen at the same time as testing. There’s no straight line running between the first pipeline and the last one on the list.

For that reason, rather than thinking of your continuous delivery tool chain as a series of pipes that are all connected serially, think of the delivery chain as one big pipe into which multiple smaller pipes feed.

Microservices and continuous delivery tool chains

Another important factor worth bearing in mind is that, in a microservices and container based environment, you may have multiple pipelines of the same type spread between your different microservices. For example, there might be several coding pipelines, each one associated with a particular microservice.

At the same time, though, other pipelines in a microservices environment could involve all of the microservices. To perform usability testing, for example, you’ll probably rely on a single pipeline even if you have a microservices app. That’s because users will need to experience your complete application in order to evaluate it. Usability testing on a single microservice does not make much sense if it allows users to experience only a fraction of your app’s functionality.

Integrating the continuous delivery tool chain

To build the most effective continuous delivery pipeline, you have to make sure that all of your smaller pipes feed into your delivery chain in the right order and at the right time.

This is important because your various DevOps pipelines are interdependent. If work from one pipeline flows downstream before another pipeline is ready for it -- or if it arrives late -- your workflows will go out of sync.

How do you solve that challenge? How can you optimize the arrangement of all of the small pipelines that feed your software delivery process?

The following steps can help you integrate all of your DevOps pipelines in an optimized way

Identify the independent pipelines within your software delivery chain. Ask yourself which workflows need to be complete before another workflow can begin. You’ll find that some workflows have no dependencies. For example, software architecting is often the first step in the delivery process and can be started independently of other pipelines.

Identify dependent pipelines and order them. Determine which pipelines depend on others, and the order in which they should be arranged. For instance, software architecting needs to be complete before coding can begin. Coding needs to be complete before quality assurance and usability testing can start. Arrange your delivery chain accordingly.

Identify pipelines that should run in parallel. Some processes, like the different kinds of testing, can and should happen concurrently. If they don’t occur in parallel, you waste time.

Adjust volume and scale. Some of your DevOps pipelines will involve larger teams and more effort than others. To keep the delivery chain operating smoothly, you have to make sure that the volume and scale of all of the pipelines is properly balanced. For example, if your coding workflow is a large endeavor and produces a lot of new code quickly, you have to ensure that the testing processes that happen downstream from coding are prepared to handle the volume of work that reaches them.

Keep pipelines flexible. Agility is a key part of the DevOps mindset. When designing the arrangement of your pipelines, make sure to keep the structure flexible. Maintain the ability to reassign members of one team to another when the scale of pipelines has to change. Keep in mind that you may end up adding or subtracting pipelines as your priorities change.

Conclusion

Building an effective continuous delivery tool chain requires thinking in more complex terms than just a single pipeline -- especially in a microservices environment. You have to take into account all of the individual, disparate workflows that form the software delivery process. Each of these workflows represents a distinct pipeline.

To avoid delays and maximize the overall productivity of your software delivery effort, you should make sure that each of your pipelines connect to each other -- and into the continuous delivery tool chain -- at the rate places and the proper scale.