Introduction to mobile test automation

Before we start learning about mobile test automation, let's understand what functional test automation is.

Test automation has always been a fundamental part of the software testing lifecycle for any project. Organizations invariably look to automate the repetitive testing actions in order to utilize the manual effort thus saved for more dynamic and productive tasks. Use of automation tools also allows utilization of system idle time more effectively. To address these needs, there are a plethora of tools available in the market along with various frameworks and implementation techniques. There are both open source and licensed tools available in the market. Tools such as HP's Unified Functional Testing (UFT), formerly known as QuickTest Professional (QTP), TestComplete, Selenium, eggPlant, Ranorex, SilkTest, IBM Functional tester, and numerous others, provide various capabilities for functional automation.

However, almost all of these tools are designed to support only a single operating system (predominantly Windows—owing to its popularity and the coverage it enjoys across industry verticals), although a few provide support for other lesser-used operating systems, such as Unix, Linux, Sun Solaris, and Apple Macintosh.

As far as functional automation is concerned, you don't need to even consider the implications of supporting multiple operating systems in most cases. With Windows as the only operating system that is supported, there aren't any considerations for different operating systems. If the application is a web application, then there may be a need to do cross-browser testing, that is, testing automation on various browser types (Chrome, Firefox, and Safari besides Internet Explorer) and their respective versions.

Also, as far as functional automation is considered, there is a very clear demarcation between nonfunctional and functional requirements. So, an automated solution for functional testing is not required to consider factors such as how others processes running on the machine would impact it, or any of the hardware aspects, such as the screen resolution of monitors and the make of the machines (IBM, Lenovo, and others).

When it comes to mobile automation, there is an impact on the test suite design due to various other aspects, such as operating systems (Android, iOS, Blackberry, Windows) on which the application is supposed to be accessed, the mode of access (Wi-Fi, 3G, LTE, and so on), the form factor of the devices (tablets, phones, phablets, and so on), and the behavior of the application in various orientation modes (portrait, landscape, and so on).

So, apart from normal automation challenges, a robust mobile automation suite should be able to address all these challenges in a reliable way. In the coming chapters, we will learn about how to create such frameworks, but before that, we need to have a thorough understanding of the challenges.

Fragmentation of the mobile ecosystem is an aspect that compounds this manifold problem. An application should be able to service different operating systems and their flavors provided by original equipment manufacturers (OEMs), such as Apple with iOS, Google's Android with Samsung, HTC, Xiaomi, and numerous others, Windows with Nokia and HTC, and even Blackberry and other lesser-used operating systems and devices. Add to this the complexity of dealing with various form factors, such as phones, tablets, phablets, and their various hybrids.

The following figure is a visualization of the Android market fragmentation over various equipment manufacturers, form factors, and OS versions:

As we know, test automation is the use of software to automate and control the setting up of test preconditions, execution of tests, test control, and test reporting functions with minimum, or ideally zero, user intervention. Automating the testing for any mobile application is the best way to ensure quality, and to achieve the quick and precise results that are needed to accommodate fast development cycles.

Organizations look toward functional test automation primarily to reduce the total cost of ownership over a period of time, and to ensure the quality of the product or application being developed. These advantages are compounded many times for mobile test automation and hence it provides the same advantages, but to a much greater degree.

The following are the various advantages of mobile test automation for any project:

• Improved testing efficiency: The same scripts can be used to run uniform validations across different devices, operating systems, and application types (of the same application), thereby reducing the test execution effort considerably. This also means that the return on investment (RoI), which typically takes about 3-5 cycles of executing the conventional functional automation to achieve breakeven, is viable in most cases within the first release itself, as mobile testing is typically repeated on many devices. So, in this case, fragmentation acts as a positive factor if the automation is employed properly, whereas, with pure manual testing, it greatly increases the costs.
• Consistent and repeatable testing process: Human beings tend to get bored with repetitive tasks and this makes such a test prone to errors. Due to the effect of fragmentation in the mobile world, the same application functionality needs to be validated across various combinations of operating systems, application types, device manufacturers, network conditions, and many more. Hence, the use of automation, which is basically a program, ensures that the same scripts run without any modifications every time.
• Improved regression testing coverage: The use of automation scripts allows the regression tests to be iterated over multiple combinations of test data. Such data-driven scripts allow the same flow to be validated against different test data combinations. For example, if an application allows users to search for the nearest ATMs in a given area, basically, the same flow would need to be tested with various zip codes as inputs. Hence, the use of automated scripts would instantly allow the test coverage to be increased dramatically.
• More tests can be run in less time: Since automated scripts can be run in parallel over various devices, the same amount of testing can be compacted inside a much smaller time window in comparison to the manually executed functional testing. With the use of automation scripts that include device setups as preconditions, the execution window can be exponentially reduced, which otherwise would take a manual tester considerable time to complete.
• 24/7 operation: Although any functional automation suite can lead to better resource utilization in terms of executing more number of scripts in lesser time, with respect to mobile automation, the resources are often expensive mobile devices. If functional testing is done manually, then more of the same devices need to be procured to allow manual testers to carry out tests, and especially, more so in the case of geographically distributed testing teams. Mobile automation scripts, on the other hand, can be triggered remotely and can run unattended, reducing the overall cost of ownership and allowing 24/7 utilization of devices and tools.
• Human resources are free to perform advanced manual tests: Having automation scripts perform repetitive regression testing tasks frees up the bandwidth of manual testing teams for exploratory tests that are expensive to automate and cumbersome to manage. Hence, the use of automation leads to a balanced approach, where testers can perform more meaningful work and thereby improve the quality of delivered applications. In mobiles, since regression is more repetitive on account of the fragmentation problem, the amount of effort saved is manifold, and hence, testers can generally focus on testing aspects such as user interface (UI) testing and user experience testing.
• Simple reproduction of found defects: Since automation scripts can be executed multiple times on demand and are usually accompanied with reports and screenshots, defect triangulation is easy and is just a matter of re-execution of automation scripts. With pure manual testing, a tester would have to spend effort on manually recreating the defect, capturing all the required details, and then reporting it for defect tracking. With mobile automation, the same flow can be triggered multiple times on a multitude of devices hence, the same defect can be replicated and isolated if it occurs only on a specific set of devices.
• Accurate and realistic real-life mobile scenarios: Since a mobile requires tests to be specifically designed for variable network conditions and other considerations, such as device screen sizes, orientation, and more, which are difficult to recreate accurately with pure manual testing effort, automation scripts can be developed that accurately to recreate these real-world scenarios in a reliable way. These types of tests are mainly not required to be developed for functional automation suites, and hence, this is one of the major differences.

For the most realistic results, conventional wisdom is to test automation on actual devices—without optical recognition, emulation, jailbreaking, or tethering. It is impractical to try to automate everything, especially for mobile devices. However, leveraging commercial off-the-shelf (COTS) tools can vastly reduce the cost of automation and thereby enhance the benefits of the automation process.

In the following section, we will discuss in detail the challenges that make mobile automation vastly different from conventional functional automation.

The following are some of the issues that make the effective testing automation of mobile applications challenging:

• Restricted access to native methods to enable automation tools: Traditional functional automation tools utilize native operating system methods to emulate user interactions. This is comparatively easy to do as the operating system allows access. However, the same level of access is not available with a mobile operating system. Also, inter-application interactions are restricted in a mobile operating system and each application is treated as an individual thread. This is normally only allowed when a phone is rooted or when the application under test is modified to allow instrumentation access. So, using other software (the test automation tool) to control user inputs in a mobile application is much more difficult to achieve and consequently slower or more error prone.

  For example, if an Android application under test makes a call to the photo gallery, then the automated test would not be able to continue because a new application comes to the foreground.

• Lack of prediction techniques for UI synchronization in a Mobile environment: In addition to the restricted access mentioned in the previous point, mobile application user interface response times are dependent on many variables, such as connection speed and device configuration other than the server response times. Hence, it is much harder to predict the synchronization response in a mobile application. Due to this automation of mobile, the application is more prone to be unstable unless hardcoded wait times are included in the automation scripts.
• Handling location-specific changes in the application behavior: Many mobile applications are designed to interact with the user location, and behave differently as per the change in GPS coordinates. Since network strengths cannot be controlled externally, it is very difficult to predict the application behavior and to replicate the preconditions of a network strength-specific use case through the use of automation. So, this is another aspect that every automation solution has to address appropriately. Some automation tools allow the simulation of such network conditions that should be specifically handled while developing the automation suite.
• Supporting application behavior changes for varied form factors: As explained earlier, since there are different screen sizes, available for mobile devices, the behavior of the application is often specific to the screen size owing to responsive design techniques that are now quite widely used. Even with the change in the orientation of the devices, application use cases have alternative behavior. For example, an application interface loaded in the portrait mode would appear different, with objects in different locations than they would appear in the landscape mode. Hence, automation solutions would need to factor this in and ensure that such changes are handled in a robust and scalable way.
• Scripting complexity due to diversity in OS: Since many applications are developed to support various OSes, especially mobile web applications, it is a key challenge to handle application differences, such as mobile device input methods for various devices, as devices differ in keystrokes, input methods, menu structures, and display properties. With different mobile operating systems in the market, such as Android, iOS, Brew Symbian, Tizen, Windows, and BlackBerry (RIM), each having its own limitations and variations, creation of a single script for every device is a challenge that needs to be adequately tackled in order to make the automation solution more robust, maintainable, and scalable to support newer devices in future.

In subsequent chapters, we will discuss how to effectively manage these using various techniques.