How realistic is your mobile testing strategy?

From the fragmentation of the mobile market, to the unpredictability of the environments in which applications might be accessed, there is no shortage of challenges associated with developing, deploying and managing the application lifecycle of mobile software, and that means organizations must take the idea of a mobile testing strategy seriously. While not every possibility can ever be accounted for, part of the due diligence of any organizations embarking upon a mobile strategy is to ensure that their programs have been tested in as many realistic situations possible, be it on low latency networks, or on devices that are severely underpowered in terms of memory or processing power.

You need to be able to simulate real end user experiences under typical loads during peak hours. 

Eran Yaniv, Perfecto Mobile

The world of mobile development is pushing the capabilities of today's IDEs and testing suites to their limits and beyond. There are certainly a variety of factors that make mobile applications difficult to design, develop, and test. If an organization is thinking about their mobile test strategy, here are just a few of the important topics that absolutely must be addressed:

• The fragmentation of mobile necessitates code revision for a seemingly endless array of operating systems and devices
• Hybridization can greatly increase the complexity of design for software when development teams have little experience with native code compared to HTML5 (or vice versa)
• Apps are accessed in a wide variety of different geographic locations which means there is a lot of variation in the quality of network access
• Something as simple as battery charge or the browser being used can impact app performance
• Other applications and features on a device may affect how the app reacts
• Users interact with software on a handheld device differently than they do on a desktop/laptop
• Tolerance for latency is extremely low for "on-the-go" end users
• Scalability poses a significant problem due to spikes in traffic

How can an organization tell if their new mobile app will have a successful launch or a disastrous debut? They should begin by expanding testing parameters to capture as many different scenarios as possible.

Consider networks and loads during testing

As CEO and founder of Perfecto Mobile, Eran Yaniv has seen many clients oversimplify mobile testing by assuming that all networks are created equal. Testing applications on various types of networks such as wifi, 3G, and LTE is essential. However, it's still just the start. "It isn't enough to test for different network operators and locations in the world. You also need to be able to simulate real end user experiences under typical loads during peak hours. This doesn't necessarily mean extreme conditions like the SuperBowl or Black Friday. It could be rush hour while people are driving back home. This might be when mobile applications get slower and have performance degradation." Testing against network types and simulating loads for specific hours gives developers a chance to more fully understand the end user's experience.

There are some general steps that development teams can take to solve performance issues when an underlying problem has been identified in testing. Eran mentioned browser caching and local storage as potential solutions when network conditions are less than optimal or when heavy traffic might make an app less responsive than normal. The key is to do performance testing as soon as possible. If possible, unique transactions should be integrated into the testing cycle to reveal hidden issues.

For example, a simple login transaction might be tested with simulations of different network bandwidths. The start and end times along with details from the packet network file can then be inspected to see when and where things slow down. It might be the network, or it might be a bug in the code. Either way, it's critical to know as soon as possible.

While mobile testing with "device clouds" and network simulators is a necessary foundation, there are ways to take things even further. While virtual testing environments can mimic devices, operating systems, browsers, and networks, they don't reveal one of the most important aspects of mobile—how end users will respond.

This is a niche that's being filled by crowd-sourced testing. Ray Solomon, co-founder of uTest (soon to be renamed "Applause"), described how mobile testing "in the wild" works. "It's important for the app to be tested by the right people. We allow anyone to sign up, but not everyone is selected to participate in testing. We match testers with applications based on their skill set, type of device, experience level, hobbies, interests, location, and other information. This ensures that they offer feedback that is relevant to our business clients such as information about coverage, usability, and performance."

What's the benefit for the enterprise? Crowd-sourcing may prove to be a particularly effective way to test different battery states on devices and determine how real users interact with an app. Roy shared that scale resolution, CPU and memory consumption, and even battery hogging can all be revealed through crowd-sourced testing and corrected before the formal rollout date. If real users say a piece of mobile software is ready for launch, a business may have more confidence that there will be no unexpected and unpleasant surprises when the app goes live.

What are your best practices for testing mobile applications? Let us know.