Where are the REST alternatives for real-time applications?

Developers face several challenges in building responsive mobile and browser applications that synchronize data across thousands or even millions of users or IoT devices. This used to only be an important consideration for massively multi-player games. But the success of Uber's application for displaying car location's to thousands of mobile users has encouraged many enterprises to consider how to leverage the same principles in new application.

Developers have grown comfortable with writing Web applications using RESTful interfaces to push data out to mobile applications. While this makes for easy development, these are not necessarily the most performant applications, since it relies on the chatty TCP protocol underneath, and battery eating publish subscribe mechanisms for pulling in updates. Developers end up writing a lot of glue code instead before sitting down to writing the interesting part of their application, said Matt DeBergalis, co-founder of The Meteor Project.

The Importance of New Protocols

Developers are now starting to explore a variety of novel protocols that use UDP instead of TCP for pushing and pulling data between users and devices including WebSockets, MQTT, CoAP and DDP. At the same time, it is important to leverage a protocol that makes it easy for backend developers to expose real-time APIs in a simple standardized way.

As more enterprises build real-time APIs based on ad-hoc protocols, the inability to easily interoperate becomes a big challenge 

Slava Akhmechet, Founder of ReThinkDB

WebSockets make this technically possible but doesn't expose a good framework for real-time APIs in a way that REST does for non-real-time ones. Meteor's DDP specification is another good start, but has not been widely adopted yet. It provides a standard way for real-time applications to inter-operate, much like REST does for Web applications today.  "As more enterprises build real-time APIs based on ad-hoc protocols, the inability to easily interoperate becomes a big challenge," said Slava Akhmechet founder of ReThinkDB.

Other protocols like CoAP and MQTT could grow in importance, particularly as enterprises begin weaving data from the IoT into real-time applications, said Ilan Sehayek, the CTO of Jitterbit, an agile cloud integration solution. They will not replace enterprise messaging standards like JMS, but they will be an area of focus for several verticals, especially manufacturing. There are simple ways to bridge CoAP or MQTT and REST so developers used to RESTful interfaces won't necessarily need to learn the ins and outs of these protocols immediately, unless they are focused on delivering optimal performance.

Building a real-time UI

In addition, there has been tremendous innovation on the front-end to make building real-time apps like these possible. Frameworks like Meteor, Angular, and React treat real-time use cases as first class citizens, and all major web browsers support technologies like WebSockets and long-lived HTTP connections to make pushing data to the browser in real-time fast and easy.

Sophisticated teams with big budgets and lots of time can implement this functionality on top of existing databases using a combination of traditional features such as views, triggers, and third-party queuing systems like RabbitMQ. That can work quite well, but is very challenging and expensive to build because this approach is very error-prone and takes a lot of development resources, said Akhmechet.

The biggest challenge with building these kinds of apps is in the data layer on the backend. All databases to date have been designed to respond to queries, not to push data out to the back-end when something changes. Back-end engineers are forced to build complex infrastructures and sophisticated custom code to work around these limitations in the data layer. Much of this work requires custom code on the front-end, which really further complicates the development and deployment process.

Add a layer of abstraction with synchronized databases

New kinds of data infrastructure are emerging to tackle this problem like Firebase and RethinkDB. They push data to the application and take care of scalability issues, making real-time app development dramatically cheaper and far more accessible.

With Firebase the developer can just write data through a client API and respond to the changes. All the scalability and infrastructure challenges are taken care of by the Firebase service. RethinkDB is a newer open source contender. But it requires a back-end, since it cannot be accessed directly from the client. This makes it more difficult for front-end developers to get started.

"Real-time APIs for the web are relatively new. It will take some time for the industry to develop best practices and train engineers to ensure a robust healthy ecosystem," said Akhmechet

What alternatives have you found for REST based APIs? Let us know.