Software Testing Debates Part 2- Cloud and Mobility Testing

The previous article, Software Testing Debate Becomes Open Warfare, was ostensibly about Software Tester Certification but the underlying issue was about traditional vs new approaches to software testing. I suggested at the end of that article that the software testing debates are part of a larger debate about traditional vs new approaches to the whole software engineering lifecycle. I think that comment also applies to the current article, which looks at the question: Are our current software testing approaches wrong for cloud and mobility testing?

“Test is Dead”

“Test is Dead” was famously (or notoriously) the Opening Keynote Address at Google Test Automation Conference (GTAC) 2011. The Address was by Alberto Savoia, Google Director of Engineering and Innovation Agitator, and the Speakers page noted “The way most software is designed, developed, and launched has changed dramatically over the last decade – but what about testing? Alberto Savoia believes that software testing as we knew it is dead – or at least moribund …Alberto will cast stones at the old test-mentality and will try his darnedest to agitate you and convince you that these days most testers should follow a new test-mentality.”

Of course, Alberto was being deliberately controversial and some caveats emerged during question time, where he acknowledged that traditional testing approaches still have their place, e.g. for life-critical or mission-critical software. However, in this article we are looking at some more recent thoughts by practicing professionals on what needs to change for cloud and mobility testing.

Cloud Testing

A couple of August 2013 Network World articles by Ole Lensmar, Chief Architect at SmartBear Software. give an overview of current discussion and some interesting  thoughts (in some cases controversial) on both cloud and mobility testing . In What exactly is ‘cloud testing’ anyway?, Ole takes a businesslike approach to defining ‘cloud testing’ , following “several rather intense discussions”. He notes that “the major source of confusion in these discussions usually revolves around the definition of cloud itself”, so Ole starts by identifying the common traits for “cloud applications” (infrastructure, resources, access etc.) He goes on to ask if there are “any specific traits related to a cloud application that mandate extra consideration?” and notes that “My immediate answer to this question used to be ‘No.’ ”

However, “thanks to my colleagues’ persistence’”, Ole has now changed his view on this and he lists some examples of cloud testing aspects which may need special consideration (Performance, Security, Third-party dependencies), with some thoughts on what those special considerations may involve.

Ole also notes that a common interpretation of “cloud testing” is using the cloud to run or manage the tests themselves, e.g . “to generate massive distributed load tests, simulate a large number of mobile devices, or run functional and performance monitors from all over the world”  but thinks it is a stretch to call this “cloud testing”.

Ole’s article concludes with a confession. “So, did this initial insight help me in understanding what “cloud testing” is? Well, to be honest, not really. Although I do agree that there are things to be aware of when testing an application in the cloud, I still struggle with “cloud testing” being something separate than “regular” performance, integration or security testing.

All of these just need some special consideration and understanding when applied to an application running in the cloud.”

See http://www.networkworld.com/community/blog/what-exactly-cloud-testing-anyway for the full article.

Mobility Testing

Ole Lensmar also has some provocative thoughts on the need for testers to change their ways in terms of mobility. In his 20^th August 2013 Network World article When will software testing be truly mobile? Ole asks “Will testers be among the first IT professionals to shift their toolset and workflows from desktops and laptops to tablets and mobile devices?” Ole says that with his developer hat on, he can’t wait for the “monumental shift from desktop to mobile.”  His mind is constantly occupied with solving hands-on coding problems and he will be able to solve them immediately and easily on a tablet and “on the go”

Ole thinks the same should apply for testing, since software testers “seem to be a prime target for a mobile mini-revolution; not only should they be testing mobile apps, the whole quality lifecycle should be available at their fingertips on their mobile devices: test design and management, exploratory testing, test recording and execution, regression testing, etc”

However, he goes on to ask “So will it happen? Will testers be among the first IT professionals to shift their toolset and workflows from desktops and laptops to tablets and mobile devices?” then provides the answer:  “Unfortunately, probably not.”

So what are the barriers? Ole sees testing and QA as a conservative domain, “with testers as a group often slow to adopt many of the ongoing trends in development.” He also thinks “

testing as a profession and practice has generally been kept short in larger organizations and not seen as a target for investment and innovation”. Ole’s third hurdle is what he calls “the anti-tool movement” , in which “many testers feel (with good reason!) that tools hamper their creativity.” See http://www.networkworld.com/community/blog/when-will-software-testing-be-truly-mobile for the full article.

Mobile Application Testing

A different take on mobility testing comes from an article in the Apr – Jun 2013 Issue of OZTester Magazine, Mobile Application Testing – Return of Technical Tester? which  suggests that “The trend in the number of mobile devices, including smart phones and tablets, is a very significant development that will have a far-reaching impact on applications development and testing. Applications  are not only becoming mobile-enabled, they are also becoming an integral component in completing business transactions.” The author is Reghunath Balaraman, Enterprise QA Transformation Principal Consultant with Infosys, who asks if we are we seeing “a shift in focus from the domain knowledge that made testing vertical-oriented (segmented based on industry or domain) to testing becoming a horizontal offering (spanning across industries) as it used to be?”

To answer this question, the article takes a highly technical “deep dive into the technical aspects of testing in the world of mobile application quality assurance.” This covers the three broad categories of software for mobiles (native apps, mobile web and hybrid) and discusses the different testing strategies and approaches needed for each. On the use of emulators, the author notes “emulators help run the code in an emulated environment to check how it will perform when it is deployed on a real device. This is a very good tool for unit testing and to some extent, the initial sanity testing. However it can never replace a real device.”

On test automation, the author advises “Given the repetitive nature of the tests, automation is an obvious choice. However, it comes with its own challenges”, which he goes on to identify, concluding “My personal preference is to extend the existing functional automation tool rather than introducing a new tool that is aimed solely at automation of mobile components.”

In conclusion, the author suggests that  “there is an increased focus on the technical abilities of the tester and perhaps the re-birth of the ‘technical tester’. This surely will not make the domain-focused or context-driven testers obsolete. But it will emerge as a desired skill – if not establishing itself as a parallel role to the  domain-focused functional tester.”

See http://www.nztester.co.nz/oztester_1_files/OZTester01.pdf for the full Apr – Jun 2013 Issue of OZTester Magazine or http://www.infosys.com/IT-services/independent-validation-testing-services/Documents/technical-tester.pdf for a reprint of the article.

Use of software testing automation tools, mentioned by both of the above authors, is also a hot topic in the Software Testing Debates arena – but that will have to be another article.