User scenarios and Functionality may be the major theme while testing any app(mobile or desktop). Software generally comes with a detailed help file. Complex and unfamiliar workflows later become a standard. But the trend is changing in the mobile world with a minimal help file and more intuitive ways of interacting with software with a minimal learning curve. Mobile users use applications on the go, so network availability and data consumption play a major factor while testing. So, there are several factors to test for in Touch devices when it comes to mobile app testing.
This mnemonic COP FLUNG GUN is to remind testers of what to think about when they are creating tests. This is inspired by James Bach’s SFDPOT (San Francisco Depot) and Jonathan Kohl’s I SLICED UP FUN!
Communication
People may use smartphones for different reasons, but the primary task of the phone is to receive and make voice calls. How does your app behave on interruptions during incoming calls? How does your app communicate with voice calls, texting, and voice messages? How easy is it to attend, reject, hold, or disconnect a call while using your application? Test the app not just on Wi-Fi, but also with good cellular connection with constant interruptions and noises.
Orientation
People can rotate mobile devices at any time for various reasons. Whatever their reason for rotating the device, people expect the app to maintain focus on basic functionality. Do you test changing the orientation in a signup page of the app? While on a pop-up screen? With notifications? With touch keyboard on? While filling a form? Do you change orientation and scroll, swipe between pages, pinch zoom? Does the app change its size and alignment as per orientation? If the app is not designed to change its orientation, are people informed? Never test mobile applications keeping it static at your desk, carry them to the pantry, to restrooms, use it while traveling, on your bed, lying on a couch. Apart from observing orientation, also look for ergonomic related tests and clues.
Platform
One of the ways to understand the platform is to explore, observe, and learn the platform’s standard apps. Apart from those, track and use the most popular and most downloaded apps like Zyte, Flipboard, Instagram, and games like Temple Run, Draw something. These apps will help in knowing various gestures, how certain gestures can help users accomplish a task with ease. These apps will help compare usability and user experience of the apps under test.
When it comes to Apple you have to think different, Apple thinks different and set a trend, and rewrite the rules. So, heuristics like similar products can’t be applied here. So, how to test such a new paradigm shift? To test Apple products you should be a fan of Apple, a true apple fan, else you would never understand why things are done a certain way. Apple focuses on getting better ways for people to interact with software; it never wants users to worry about the internal file system.
Though heuristics and mindset need change, the core skill to test remains the observation skills. Observation starts even before you open application, even before you power on your iDevices. Have you observed the way people carry their iPads? Covered with a neat leather case makes them look as if they are real paper notebook. Have you ever observed how people take notes on an iPad? Do they click on a new file? Click on save? Search in a folder to open xxx.TXT file? You click, now tap the notes app icon, type your notes, once done close as if you were taking notes with a pen. Apple never wants end-users to learn all technical details to take simple notes. Apple restricts end-users from accessing the core OS file system, but for good.
Android, most tech-savvy people embrace android mainly because it is an open platform, not restricting users from accessing the OS. Users here expect every app to behave the way they love when they change their settings. Do you derive tests by discussing with tech-savvy android users? Do you test with unrooted phones? There are many different display resolutions, OS versions available; it’s good to have a combination matrix to cover the most popular screen size and resolution variables. Though there are simulators, they would fail on a real device. One recent example is the Temple Run game failure on many android models.
Function
Everything that a product does, the most common form of testing, for the requirements. See that each function does what it’s supposed to do and not what it isn’t supposed to do. Starting, shutting down of the apps, interactions, multimedia, file access, navigation, user interface, data syncing on the cloud, etc. Did you tap all known interfaces? Did you enter all possible data in the form? Did you change the settings?
Location
Location tracking is one of the key features to be tested in mobile applications. Unlike desktop devices, mobile devices are mostly on the move. Geo-location tracking: Most of the apps these days have the feature to track your location, how easy is it for people to change the settings? Are people informed clearly about their location tracking? Does your application adhere to mobile platforms location tracking and other privacy guidelines? Moving from one location to another location how is the application behavior when moved out of Wi-Fi connection? Moved from one data network to another? Never test mobile applications sitting in one place.
User Scenarios
Mobile app testing is about thinking about real scenarios of how a mobile device and the application would be used by people. Try to think of how a tech-savvy user would try to accomplish a task. Think about how a novice user who never had a smartphone would try to accomplish a task. Try credible, complex user scenarios. List out possible users, environments, commonly used scenarios, extreme scenarios, disfavored scenarios.
Network
Mobile applications are mostly used on the move, and so depend on availability, strength, and reliability of Wi-Fi and cellular networks. How does the application respond while moving between Wi-Fi, GPRS, and 3G? Does your application automatically use Wi-Fi when available instead of a cellular network?
Gestures
In this part of the world, apps respond to gestures, not clicks. People expect all the gestures to work the same, irrespective of apps. Does your application respond to gestures? Navigation is by swiping on the screen, no more, next and back buttons? Are people indicated properly that there is more than one page available to swipe? Does your application respond to various multi-touch gestures? Does your application use standard gestures? Are the gestures consistent across the app?
Guidelines
Most mobile platforms have guidelines for developing an app, Apple has iOS Human Interface Guideline (HIG) and Android has Design. These guidelines will help to build standard gestures, icons, and menus in a specific platform.
There are guidelines for app store approvals like Apple’s App Store Review guidelines. Use these guidelines to test for error handling, location tracking, user privacy, accessibility, logging, etc. There could be important changes in guidelines, which need to be tracked and tested for changes, like Apple restricting apps from accessing UDID to identify users.
Updates
Another important factor to consider in mobile platforms, unlike desktops the updates here are frequent. While testing, it’s always good to start with at least one version lower than the latest, use the app, now upgrade to the latest available version, and observe the application behavior. Updates also need to be tracked to learn about new features available to app developers, and how that could change the workflow or existing gestures. Good to have app developer licenses of various platforms and explore the developer beta seeds to get to know all the latest features.
Notifications
Notifications enable an application to inform its users that it has something for them. How does your application use notifications? How easy is to turn on\off your app from notifications? Does your application use local or push notifications? Notifications in iOS are delivered via Wi-Fi only if there are no cellular networks. How does your application behave if the device is in sleep mode? How are the local notifications displayed if the screen is locked? Does the app provide too many notifications? Test for all available types of visual notifications, sound, and vibration.
How does your application behave during notifications from other applications? Test your application with all possible notifications from different apps enabled. Does your application continue to function properly after clearing notifications?
I start by measuring how quickly I could accomplish a task that an app is intended for. The measure is both in terms of time and number of taps. No user wants to spend more than a minute in a mobile device to perform basic tasks on any app. Then move on to test more complex tasks, change the setting, move between different networks, different gestures, then other mnemonics. But this is just a start, don’t restrict yourself to being a steward of received wisdom; be the author of your own wisdom” from Lesson 47 of “Lessons Learned Software Testing” book.
† – Thanks to Parimala Shankaraiah for helping to form COP FLUNG GUN mnemonic.
11