When you are deciding how you are going to build out the testing process for your mobile application, one of the first things you have to decide is if you are going to use physical devices, virtual ones (emulators, simulators), or a combination of both.
Below, we will breakdown the benefits and drawbacks to each, and lay out use cases where one may be a better fit than the other.
Benefits of Real Devices
Proximity to end-user experience
There are certain realities your end user may face that emulator or simulator devices struggle to replicate. Having a real device ensures that you are testing how that device responds to a human swiping or tapping on a screen, accounting for the accuracy (or lack thereof) of a real person.
Real-world testing environment
While a controlled environment has its benefits, providing a real world testing environment ensures you have conditions that reflect the ones the device will actually operate in. Device hardware, sensor based testing, network and connectivity conditions, performance when the device is at low battery…all of these things will be easier to test on a physical device than a simulator or emulator.
Drawbacks to use of real devices
The set-up and maintenance
When you are using a physical device, ensuring a fresh device state after every test is your responsibility! This leaves a lot of room for human error as testers have to ensure information is not cached in the device, settings are correct before each test, etc.
It can be extremely difficult to ensure the app will behave as a true, first install version of itself on your testing devices.
Scaling costs
Supporting a wide variety of iOS and Android devices is great for your customer…expensive if you are invested in physical device testing. You are probably on the hook for multiple devices of every configuration you support in order to ensure test coverage in a timely manner. Additionally: your test results have to be stored somewhere which means server costs will scale with your testing needs.
When to leverage a physical device
- Checking performance while other apps are in use on a phone
- Verifying that certain device settings or configurations do not impact build performance (connection, battery level, etc.)
- When looking to get a feel for animations, or overall smoothness in the app as your end user will experience it
Benefits of simulators and emulators
Simplified setup and access
Virtual devices are generally easier to set-up (especially if your infrastructure provider ensures fresh devices for every run) and your provider will also keep the available device configurations up to date: ensuring you have access to the latest make and models.
Faster testing and scalability
Virtual devices can run tests on multiple configurations at once…as well as run multiple tests simultaneously. This results in way faster testing times! You can easily increase/decrease the number of devices you test on based on your needs.
Optimized for cost efficiency
Since you are not purchasing physical devices in the quantity required to fulfill testing needs, you generally see a cost savings when using virtual devices. Shared infrastructure packages, where the devices are not owned by your account alone, are a popular option for cost effective device access. Dedicated, virtual devices are also offered by most major providers.
Controlled test environment
Simulated devices offer a completely controlled test environment: you dictate the conditions of the test, and they cannot be impacted by the real world in unexpected ways. This helps open the possibility of mobile test automation, as the controlled environment ensures repeatable test conditions.
Drawbacks to virtual devices
Replicating certain real-world conditions
At the end of the day: simulated devices may struggle to exactly replicate conditions or behaviors that reflect an end-user experience. Replicating network conditions, or even accurately reflecting a user interaction on a screen can at times present challenges.
When to leverage virtual devices
- When you need to scale test coverage
- When you need to test faster
- When you are ready to begin implementing automation
In conclusion...
Overall, both simulators/emulators and physical devices have roles to play in testing. However, when it comes time to scale your testing coverage, adopting virtual devices for at least a portion of your testing needs is a wise, resource efficient move.
Waldo offers users access to a fully maintained testing infrastructure that includes fresh devices in the configuration of your choosing for every test run!
Reach out to our team today to learn more about how we can support your testing needs, or try our free manual testing tool (Sessions) for a sneak peak of the platform.