(In this SmartBits, Shivaji Raju outlines “ Digital test automation“. The video is at the end of this blog)
There is greater focus on services testing with lot of applications being built with service based architecture, that is one of the things that is changing significantly. From a framework standpoint, there are approaches like BDD and some custom Java-based solutions that people are trending to use when compared to the traditional keyword driven approaches that we were using in the past. The scale at which we test on different types of devices has also increased vis-a-vis when we used to test only on one browser, like IE. Now, the expectation is that we test the product on the web, on native, on different types of devices, the combinations increases.
The other change is in terms of using testing platforms. We don’t want to host mobile devices on premise. We test on Sauce Labs, Perfecto, or LambdaTest, these are some custom solutions that are available which can be used to scale up testing. Even lot of projects have moved from waterfall to Agile based implementations.
The other big thing is the DevOps. In context of testing, continuous testing is something that needs to be automated. So, other than we automate our execution aspects where we automate smoke,regression or different types of tests, whether it is UI services; we need to also ensure that the infrastructure and the test data elements are also automated. When we say Infrastructure, can we automate the deployment of builds into test servers or can we provision the instances on the fly instead of having some manual dependencies to ensure we get an end-to-end continuous pipeline.
(In this SmartBits, Shivaji Raju outlines ” Key trends in automation “. The video is at the end of this blog)
Key trends in automation are significant focus on services testing with the architecture changes coming in. With service-first approach and microservices , there is a lot of emphasis on testing the services. The other aspect is that traditionally we had a lot of test cases or automation scripts on the UI. That seems to be changing. We are trying to bring in quite a balance between different layers of testing rather than just focusing on UI.
The other trend is on the tools. We were predominantly using tools like HP UFT, Rational tools, or Selenium to some extent. The trend now is shifting towards script-less solutions like Tosca, Functionize, Mabl which has ability to build scripts faster. Some trends have been noticed on the framework front too. We traditionally had QA-driven approaches which were quite heavy. Now the shift is to use lightweight approaches or frameworks, especially in the Agile context and frameworks that integrate to the toolsets as part of the DevOps pipeline itself. The need is to ensure that the framework integrates with different build tools like Maven, JIRA or Artifactory. Those are some of the expectations when building a solution for the framework.
Again, DevOps is a significant trend now, so the expectation is to see how we could automate the continuous testing pipeline, considering that it’s not just about automating the execution piece, automate even test data or probably automate the infrastructure piece. So for provisioning infrastructure from the clouds, there are certain tools to do that and then there are trends around virtualizing services, even virtualizing databases also to ensure that some automation is brought into the data and the services layer and that is how we would achieve an end to end CI or CT automation.
Summary The act of testing is a scientific exploration of a system done in three phases – RECONNAISSANCE to understand and plan, SEARCH to look for issues, REST&RECOVER to analyse and course correct. To enable the various activities in each phase to be done quickly and effectively, is where the SEVEN Thinking Tools outlined in this article help. How to apply these tools in a session-based approach is also briefly outlined.
When I hear people talking about testing as Manual or Automated with the latter being the need of the hour, I am flabbergasted. All the word ‘manual’ conjures in my brain is that of me doing a menial job of painful scrubbing!
It is time we used Intellectual & Tool-supported. “Think well. Exploit tools to do.”Enough of rant!
In current times, speed is everything, right? What can we do to test quickly ? Use tools. Automate. Right? Wait a minute – This is about execution, right? What about prior activities?
To answer, let us ask the basic question what is testing after all? Testing is exploration. Let me correct it. Testing is scientific exploration. And exploration is a human activity that is aided by tools & technology. How can we do scientific exploration rapidly? By using tools that help us think better and do faster.
Let us say you want to explore the nearby mountain range by foot. Would you just pick up your backpack and go on? I bet not, unless it is a really short trip. Otherwise I think you will study the geography/terrain, read others’ experiences, do a reconnaissance, create various maps of terrain, of pit stops, of food joints etc before you chalk out the full route. Once the route is setup, you will pack your bags and go. As you explore, you will discover that “the map is not the terrain” and be taunted, surprised, challenged and you will learn, adjust, improvise, revise the maps, routes as needed. Tired, you will rest, analyse, replan & recover to continue on your journey. This is not ad-hoc nor driven by sheer bravado. This requires logical thinking(scientific), planning and ability to observe, adjust continuously and also some bravado and good luck!
This is what we can apply too in testing our software/systems. This article distils this and provides you with SEVEN THINKING TOOLS to enable you do these easily and scientifically.
Applying the above analogy, we look act of testing as being done in THREE phases: RECONNAISANCE, EXPLORATION, REST&RECOVER.
RECONNAISANCE : Do survey and create maps Survey : Get to understand the system under test by reading documents, playing with the software/system, discussing with people, to clearly understand who the end users are, what the entities (e.g. features, requirements..) to test are, what the various attributes the end users expect are, and the environment in which it will be deployed. In a nutshell we want to know the Who, What-to-test, What-to-test-for and Where. This is done by the Tool #1 “Landscaper”.
Create Maps: Now that you know the key information, connect them to four useful maps: Persona map, Scope map, Interaction map and Environment map.
(Tool #2) Persona map : A list that clearly connects the “Who” to What”. This helps us understand who uses what and therefore helps us prioritise testing and certainly enables us to get user centric view to validation.
(Tool #3 )Scope map: A list that connects the ‘What’ to ‘What-for’. This helps us to understand what the expectations of the various entities are i.e. That for even feature F1, we have an expectation of performance. What does this help us do? This helps us identify the various types of tests to be done.
(Tool #4) Interaction map: No entity is an island i.e. each entity may affect one or more entities. i.e a feature F1 may affect another feature F2 and therefore a modification of F1 may require retesting of F2. How does his map help us? Well, this helps plan our regression strategy intelligently.
(Tool #5) Environment map : This lists out the various environments on which the final system may be run so that the functionality and attributes may be evaluated on various deployment environments.
Now that we have done done the reconnaissance, we should have good idea of the system under test and therefore be ready to explore.
SEARCH : Now that we have the maps, the next step would be chalk out the routes and then we are ready to commence our search for issues. This is done using the “Scenario creator” tool. Once this is done we commence our search for issues. When doing this we will encounter things we don’t know, things that we did not anticipate, issues and therefore will need to revise and course correct revise the landscape, maps and routes. This is accomplished via the Dashboard tool in the Rest&Recover phase.
(Tool #6) Scenario creator: This tool helps to design the various test scenarios that would serve as the starting point. Note that these will be continuously revised as we explore and gain a deeper understand of the system and its context and usage. What is important would be segregate the scenarios into Levels so that the test scenarios are focused and clear in their objective. Robust Test Design approach of HBT helps you design scenarios that may be done using a mix of formal techniques, past experience, domain knowledge, context but clearly segregated into various HBT Quality Levels.
REST& RECOVER: In this phase, the objective is to analyse the exploration phase results and improve what we can do, track progress of doing and judge of the quality of the system-under-test. This is done by the tool ‘Dashboard’
(Tool #7) Dashboard: This tool helps you to do there things : (a) judge adequacy by look at the map and route information and improve the same (b) track progress of work done by checking the what has been done vs.planned as far as routes are concerned (c) judge quality by looking at the execution outputs of the scenarios level wise.
So how do we apply this tools? We saw that these 7 tools could be used in the THREE phases of RECONNAISSANCE, SEARCH, REST&RECOVER by a session based approach “Immersive Session Testing”. Each session is suggested to be short and focused say 60-90 minutes with a session objective to one or a mix of the phases.
Note that a session could be an exclusive RECONNAISSANCE or SEARCH or REST&RECOVER on a combination of these. Why is the session time suggested to be 60-90 minutes? Well this is to ensure razor sharp focus on each on the activity done. Also a short focussed sessions allow one to get into a state of flow enabling higher productivity and enjoy the activity!
About SmartQA The theme of SmartQA is to explore various dimensions of smartness to leapfrog into the new age of software development, to accomplish more with less by exploiting our intellect along with technology. Towards this, we will strive to showcase interesting thoughts, expert industry views through high quality content as articles, posters, videos, surveys outlined as a SmartQA Digest weekly emailer. SmartBites is soundbites from smart people”. Ideas, thoughts and views to inspire you to think differently.
Signup to receive SmartQA digest that has something interesting weekly to becoming smarter in QA and delivering great products.
