How Pokemon Go needed a Kubernetes powered Java cloud
How did Pokemon Go scale so well? A big part of its scalability was the Java cloud platform that leveraged containers and Kubernetes.
At the Game Developer’s Conference in San Francisco, Edward Wu, Niantic Labs Director of Software Engineering elaborated on some of the growth pains of Pokémon Go. The app backend was written in Java combined with libGDX. It surpassed all engineering expectations by 50-times. Despite early launch problems, it managed to keep running, and provides an inspiration for a new era of augmented reality games.
Wu said a component of their scaling strategy was its heavy reliance on creating a Java cloud with the Google Cloud Bigtable NoSQL database, along with an architecture built on top of Kubernetes. It quickly evolved into the largest Kubernetes ever deployed. Pokémon Go was eventually downloaded more than 500 million times and had to support 25 million players in a shared Java cloud at its peak. Wu said, “The biggest challenge was that engineers had to create a shared environment that worked seamlessly across all players, while minimizing cheating.”
Building a Java cloud
The work on Pokémon Go started over 4 years ago when Niantic was launched as an autonomous unit within Google to create the game Ingress, which eventually attracted about 20 million users. Niantic build a dataset of over 5 million user-submitted locations and photos of interest that was used as the basis for Pokémon Go.
Niantic engineers finalized the client before the July 4th weekend with a soft launch in both Australia and New Zealand on July 5th. They included the basic capacity estimate and a 5X number. But the games popularity blew these estimates out of the water. Wu said, “We knew we had something special on hand when these were exceeded in hours.”
The challenge of shared realities
Wu believes that the technologies for implementing this large-scale coherence is available to startup. Rapid iteration of worldwide updates was implemented using MapReduce and in particular Cloud Dataflow to combine data, do efficient MapReduce shuffles, and scale the infrastructure. Niantic engineers had a build highly scalable custom query logic on top of scalable containerized instance on Kubernetes.
Most large applications need to worry about horizontal contention. But in this kind of shared environment, the big challenge is about the vertical contention that occurs when multiple people are attempting to mutate the gaming world simultaneously. The information has to be shared in a timely manner across all players in order to create a single coherent overlay on the real world. Wu explained, “If the game world changes as a result of a person’s access, it has to be reflected to all of the players near them.”
Niantic adopted Protocol buffers, a binary data format, rather than JSON to improve performance and reduce latency. Google Cloud Datastore was leveraged to provide a scalable back end database reflected the state of game play across all users. Wu explained “In order to tune the data you need a framework to populate hundreds of millions of entities and iterate in short order.”
Inspiring a bigger playing field
The game inspired users to walk over 5.4 billion miles over the course of the year as a byproduct of game play. This is a promising alternative to the wide range of games that take people out of the physical world. Wu said, “We believe these are the first generation of mass market alternative reality product. One of the most interesting lesson we can take from Pokémon Go is that technology can motivate people to action, especially when its connected to other people. We believe that people are healthier when they go outside and have a reason to be connected to others.”
It’s also important to note that this new era of augmented reality games leverages existing hardware. Wu concluded, “We believe we have demonstrated that augmented easily is as much about a shared world stage as it is about immersive hardware devices. The market reality for us is that everyone has a cell phone in their pocket.”
