Perfecting product migrations without customer disruption with Agile

In the early days of any emerging consumer technology, enthusiastic software developers often start with the consumer. But in some cases, it makes more sense to focus on creating applications for the businesses that are better suited to establishing and maintaining customer engagement. This was the case with MobiTV, which started out as a direct-to-consumer brand with a service for providing live television on mobile phones.

Rick Herman, Chief Strategy Officer at MobiTV said, "We decided to rewrite our back end software infrastructure to take advantage of the changing digital TV landscape, including faster networks, more sophisticated viewing devices, and the opening of content rights." The company spent considerable R&D investment to build out a robust media platform to deliver live TV and on-demand video for a variety of carriers and a newly emerging class of over-the-top (OTT) video service providers.

Meet the needs of business customers

The company decided to invest heaving in not just the applications, but the development process for meeting carrier needs. The actual application infrastructure leverages Java, Python, MySQL, and Cassandra on top of a Linux stack. An agile development process allowed them to keep a 2-week pace of development with minimal disruption. This was important for meeting its customer’s needs for flexibility in rolling out new services and business models.

We decided to rewrite our back end software infrastructure to take advantage of the changing digital TV landscape.

Rick Herman, Chief Strategy Officer at MobiTV

This new infrastructure had to be able to track users as they move across from Wi-Fi to a cellular network, authenticate them across devices, and store content so it could be paused and resumed across devices. MobiTV also needed to develop a robust content management system (CMS) to address the varying rights landscape. This allowed the company to transition from a direct-to-consumer brand to a business service enabling TV for leading carriers like AT&T Mobile television, Sprint television, T-Mobile, and  Deutsche Telekom television.

The service now drives TV viewing experiences on everything from set-top boxes, streaming sticks, phones, tablets, and PCs. The platform also had to easily integrate with a variety of carrier billing systems and manage content distribution services for the carrier customers.

Adopt an agile process

MobiTV keeps pace with carrier and end user expectations using an agile development process by working in two-week sprints. They make a priority list of features and function and then work their way down. MobiTV has created a cross-functional team that is divided into squads based on areas like product management, program management, engineering, QA, and DevOps. Cedric Fernandez, Chief Technology Officer said, "We've carved out the delivery organization between platform and what we call services. The services group handles the individual departments for each of our operators. They work on leveraging the output from the core platform team to deploy the custom apps for different carriers."

This approach allows MobiTV to keep a steady cadence of releases from the platform team. In some sprints, they might be a release of new content management system features, while other sprints might just improve media features. Fernandez said, "We try to hold to our regular cadence, which sets up expectations with the rest of the business. The services teams know what's coming and what they can take to the customers. I think we've been able, using this model, to bring a lot of features to market very quickly and with the type of quality we expect.

Keep downtime down

Part of the challenge with an agile development approach lies in balancing the constant pace of upgrades without taking customers offline. MobiTV does work with maintenance windows for some upgrades, and others which are seamless. Maintenance windows are announced to customer during invasive operations that require bringing a database offline. This does not necessarily mean the service goes down, but they are announced to customers, just in case a problem ensues.

Fernandez said, "We don’t like service interruptions. Even during maintenance windows we plan to make sure that traffic's not being interrupted to a large degree. But when you're doing production upgrades, anything could happen."

To reduce the impact of the maintenance windows, MobiTV uses parallel environments. They upgrade the parallel environment and then swap people over to the parallel environment once the team knows the upgrade is functioning as expected.

Create a dynamically scalable backend

The infrastructure runs on VMs which can be deployed on the carriers private infrastructure. Fernandez said, "The cloud doesn't necessarily work for us like it would work for some other companies that might also be in the over-the-top delivery space owing to the uniqueness of the business models and deployment models we support."

One of the big considerations has been the government regulations regarding where the content resides, the location where logging is conducted. Fernandez said, "When you are doing business in multiple countries, you have got host your content in many of those countries." Leveraging a VM machine infrastructure allows the core software to get many of the same benefits of scalability during peak viewing as in a fully hosted cloud service.

Leveraging multicast where appropriate

Another important consideration for TV lies in efficiently distributing content to edge nodes. MobiTV uses multicast delivery on the backhaul to edge nodes, and then unicast TCP to the actual end consumption devices. The delivery itself is all HTTP based which allows an on-demand experience and the ability to rewind live TV. They have also created a private CDN which can be used in-house by the carrier, or deployed in concert with public CDNs like Akamai or Limelight.

Fernandez said that multicast might save some money on network costs for live TV, but would complicate the head end and the network technology for handling the traffic. This approach also allows them to offer second screen integration in which a consumer is watching TV on a big screen, while related metadata drives applications on a user’s smartphone or tablet.

Going forward, MobiTV recognizes the importance of keeping ahead of the technology curve, since its large customers could afford to develop their own TV architecture themselves. Maintaining a rapid cadence of development helps them to stay ahead of carrier expectations, and is allowing them to strike up major new deals with carriers.

Fernandez said, "Our software development approach has helped to set expectations for our customers so they know what is coming."

What practices has your team adopted to maintain a rapid pace of new features? Let us know.