agsandrew - Fotolia
Marrying DevOps and SaaS for a long lasting relationship that works
Learn how CSG eased the friction between development and operations by combining DevOps with SaaS to aid their DevOps adoption.
As a large scale SaaS based customer care and billing company, CSG serves major cable providers and related industries with a solution that must routinely handle billions of external transactions. To keep the software operating in top shape, the company employs over a thousand practitioners spread across 40 development teams.
Like most organizations with a strong technology focus, this company faced the usual friction between Dev and Ops. Prugh demonstrated the chasm of frustration separating the two with a graphic. On one side, Dev was depicted craving speed while hating the environment and SR tools. On the other side of the great divide, Ops was obsessed with stability and hated code and Agile. Both parties despised CRQs, change releases, and production Ops in general.
But CSG’s VP of Software Development and Operations, Scott Prugh, held out hope that things could change. "It doesn’t have to be that way. With DevOps your company and customers can win and your path to production can be a lot smoother." But that path required courage to experiment with team boundaries and composition. Like other organizations, CSG quickly realized that it was necessary to do away with QA as a separate entity. "Having dedicated QA was creating a target rich environment for defects and de-optimizing value in the system." That was just the start of merging the team for greater cohesion.
Beginning the process of bridging the gap
Part of the issue with having a segmented organization where work is passed from one queue to the next is that the feedback loop is far too long. The cycle of prediction, action, evidence, and evaluation has a lengthy trajectory that makes continuous improvement virtually impossible. In addition, learning is "I" shaped. In other words, learning is narrowly based on a single role and not across business processes. It is unrealistic to expect Dev and Ops to fully understand how to create a more optimal product and a better user experience if they only experience the software from their side of the canyon.
Breaking down the structure of organizations within organizations was key to creating true synergy between Dev and Ops. Putting a CD pipeline in place was a step in the right direction as this structure helped create an environment that promoted more cross-discipline learning. Testers were paired together and developers began to learn to design for testing. The team began to see real improvements, with 10x gains in quality and 50% reduction in time to market. But there were still persistent issues with almost all incidents occurring outside of release and being fixed by Ops. It was time to bring accountability into the picture in a more meaningful way.
"Dev teams didn’t understand how the software was used in production and how their customers consumed it." At the same time, Ops had its own shortcomings. "Lack of engineering skills in Ops prevented improvement and encouraged duct tape solutions." A lack of shared mission promoted a lack of empathy across teams, and handoffs elongated lead times.
Marrying Dev and Ops at last
Ownership of the entire process proved critical for DevOps to function as intended. For this to happen, service delivery teams needed to build a product that they viewed as their own software rather than focusing on isolated tasks. The model CSG chose to shift toward brought Dev and Ops much closer together with the whole team being responsible for the majority of the delivery chain (design, build, test, and ops). With shared accountability, there was much more incentive for Dev to cross train engineering skills and principles, helping Ops get a better handle on fixing issues in a smarter way. With greater visibility into the daily pain that Ops suffers due to poorly designed software, Dev could also take greater responsibility for designing with Operations in mind.
The end result may be a left shift and a right shift—making Developers more likely to design with production in mind and Operations an engineering powerhouse for true continuous improvement rather than a process activity. Attaching the CI pipeline directly to the platform, moving away from infrastructure as a task, and getting daily feedback across the lifecycle are also expected to be crucial elements for making DevOps as efficient as possible. For Scott and the CSG team, this transition will likely be a work in progress through 2016 and beyond.
