T Ashok @ash_thiru

Summary

Doing SmartQA is about great mental clarity of visualising what is intended, what is present, what-may-be-missing that could-be added to enhance the experience. The intent to seek this clarity result in good questions that help us understand better and therefore test well is the objective of this article. Two tools for the mind “Landscaper” and “Deep diver” that can help are outlined here.

Prevention occurs due to good understanding. Detection occurs due to good understanding. Understanding of what is needed, what is stated and what is implemented.

Doing SmartQA is about great mental clarity of visualising what is intended, what is present, what-may-be-missing that could-be added to enhance the experience. The intent to seek this clarity is what one drives to question well, build better, prevent and detect issues.

The act of testing is really discovering what-should-be-there but-not there, what-is-there but not correct, what-should-be-there but should-not-be-there. Finally it is about understanding the impact of something that been changed, be it in the system or outside the system.

The key to this is understanding the product/application/solution. Understanding from different points of view- end users, construction, technology, environment, development & deployment. 

Smart Understanding is about scouring the ‘landscape’  to understand overall context and the static structure of how it is built and then ‘deep-diving’ to understand  the intended dynamic behavior. Landscape and Deep dive are great tools for the mind to explore the system rapidly to do SmartQA. The associated picture illustrates these two thinking tools well. 

Tool#1 Landscaper

Given an entity to test, be it a small component or a big business flow or the entire system, the first thing to do is examine it well by performing a ‘landscape’ . 

1. Start with understanding who the end users are, then identify the entities (the system offerings), the connect to who uses them and how much and then go on to figure out as to what they may expect from these. 
2. Now switch to a deployment view to understand how is deployed and other systems it is linked with.
3. Now from the construction/development point of view understand what is built new (via fresh code or glue code that integrates components) and what is being modified. 
4. Continuing to look from the structural view, understand how these entities may be coupled, to appreciate the interactions and what may be affected due to the new/modified entities.
5. Go a little deeper to understand the system is is architected and technology stack(s) used to implement this.

What we are doing is doing a tour from different points of view and attempting to understand the ‘whole’. During this process many questions arise which enables better clarity of problem.

Tool#2 Deep diver
Having a good holistic picture of the system under test, it is only natural to to dive deeper to understand in detail an entity. Deep diver is the second tool that helps you do this.

1. First understand the various inputs to this entity. What are they, where do they come from, what is their format and spec, rates and volumes too? Are there any interesting values ?
2. Next, understand what the various outputs are and may be. What are the normal outputs and what are those in situations of error? Do check if ‘all’’ possible outputs have been indeed been considered.
3. Finally it is time to understand the intended behavior by discovering conditions that transform the inputs to intended outputs. Note that some of the behaviour conditions could be based on system state and not based on inputs only.

Smart understanding is key to doing less, doing well and accomplishing more. This is a seriously mental activity and doing it well has great paybacks! The two mental tools “Landscaper & Deep diver” enable a logical approach to decomposing the system(i.e. problem) well so that you have great clarity on what, how-to-do and how-much-to-be-done.

In this smartbits, Girish Elchuri outlines about Design for testability.

Another practice that I follow is, during the testing times, the code is made to behave differently, though I am not altering the functionality. For example, in a particular workflow, I add a mobile number or an email id and I validate them later. There may be certain functionalities that would be possible only when you verify the mobile number and email id. These are the mundane things for which you cannot do the test automation. 

So what we do internally is when we are in the test mode and as soon as the mobile number is added or the email is added, we treat them as verified, we put that flag, so that we can do further testing in a much more efficient manner. That is just an example of how you can make the product behave differently for testability which is a much more efficient way of doing it, this is the first aspect.

Secondly, in a workflow-based product, you do certain things and for the third step of the workflow, you might need some information that is generated in the first and second steps. Normally, this will be visible to the user on the GUI but then on GUI-less test automation, it becomes difficult. So what we do is, in the testing mode, we actually generate this data as a file and then using the data which is generated as a file, we do the next steps in the workflow. This is another way that we make the code behave differently in a testing mode.

Thirdly, in a small percentage of cases, when the test is being run for subsequent steps, the code itself generates the test automation scripts. So what I do is instead of just outputting the data, I output a callable script call into the file and in the end, I just execute that file.

Since all these behaviors occur during testing, we can have a flag there saying “I am testing” and then what we can do additionally is make sure that these things are not run in production mode. These are some additional checks that we are doing. Again, that is where I see a developer helping the testing folks by putting this additional code to facilitate the testing.

In this smartbits, Raja Nagendra Kumar outlines as to what it takes to innovate.

Innovation can be classified as fresh Innovation, which does not exist or you have to make it better than what it is now.  In the context of clean code and technical debt, most of the Innovation can come from trying to make things better than what it is. A product can do billions of dollars of business. And if you are not innovating, we say few billions are actually hidden if you know how to innovate within what is existing, there are few more billions can get added because it could be that you may find new features, new way of doing it or you are able to run faster than the competition and acquiring more customers faster. 

Any product,  makes money by people adopting it and if you can reduce the customization time and improve adding more features, there are hundreds of features can come in and one of the techniques, I follow for this kind of innovation is to keep the code clean. Most of the time, because people are living with technical debt, patchwork,  being inefficient, they are not able to see what can be done better. Most of the time, there is so much of code so much pressure. They are not able to see what can be innovated and if you start simplifying this baggage, automatically you will see tons of innovative patents that can come in from the same product.

Sun Microsystems had one of the best products to be sold as a hardware company. They tried to be like a software company, but most of the time, because they got lost on the fire the solution itself got lost somewhere. There could have been a better way of solving it. We could have improved adaptability and reduced support ticket. Later we refactored unit testing and eliminated technical debt. We were able to find a better solution and that was patented and it is a patent now. 

That way most of the time, people living in a product world, if they know how to simplify or remove the technical debt and automatically it starts becoming simpler and then they will start questioning the solution. And this solution can give a lot more innovative ideas. 

T Ashok @ash_thiru on Twitter

Summary

This article outlines twenty approaches to smart test design based on seven views of user, logic/analysis, construction, test, experience, operational and evolution.

Let us talk a bit of test design now. We focus a lot of execution, and therefore the ability to cover more. The focus has veered to how frequently we are able to execute the tests and therefore on automation. Let’s step back and ask to what the objective was. Well it was to primarily deliver clean code. So we need to have a deeper sensitivity to the quality of tests. This is where design becomes important.

So what is a smart way to design to come with good scenarios, ideally few that can uncover issues that matter most.

Smart Test Design is about looking at the system from multiple views to:

I want | expect | would-like behaviours to satisfy needs that are implemented well and comprehensively covered to help me do well on my environments with no side effects.

Then decide what you want to prevent, statically detect or test, be it via human or a machine. Focus on intent and then the activity.  

Let me list down TWENTY approaches to smart test design looking at the system from SEVEN views.

1. User view based

1.1 Use the requirement specification to design.
1.2 See actual users of how they work and then use this to design.
1.3 With users doing experience sessions with system and use this information to design.

2. Analytical view based

2.1 Use software test techniques (black or white) on spec/structure to design.
2.2 Construct behavior models and use this to design. 
2.3 If the system needs to comply with a standard, use the standards information to design. 

3. Construction view based

3.1 Use the code properties like lines/conditions/path to design.
3.2 Exploit your deep understand of technology to understand potential mechanisms/flaws and use this design.
3.2 Understand how the system has been architected, composed & integrated to design. 

4. Test view based

4.1 Hypothesise potential faults probable given your understand of usage, structure, architecture, environment, conditions to design.
4.2 Use potential error return codes, exceptions, deliberate bad inputs, violations of system states to design.
4.3 Identify potential end failures and failure modes and use this to design.
4.4 Explore the system to understand its behaviour in various contexts and use this to design .
4.5 Probe the system with a series of questions (say what-if) and use this to design.

5. Experience-view

5.1 Use the past history of issues encountered with various customers and design.
5.2 Apply the learning of various situations or deeper knowledge to come u with fault patterns and use this to design. 

6. Operational-view

6.1 Use the understanding of actual business flows, usage profiles of features on various environments to design.
6.2 Identify various deployment configurations and use this information to design.

7. Evolution-based

7.1 Use the information of what code has been changed, to understand how these may propagate and may cause issues to design.
7.2 Use the information of changes in the environment to understand their impact on the system and design 

Smart Design is about:

I want | expect | would-like behaviours to satisfy needs that are implemented well and comprehensively covered to help me do well on my environments with no side effects.

In this smartbits, Anuradha Biswas outlines as to what it takes to make an software application accessible.

This is what she says:

“There are two parts legacy solutions and new apps and nowadays everything is about mobile apps. There are clear guidelines given out by different companies and some of it is universal like section 508 W3C WCAG and companies like Microsoft and Apple have something more specific to themselves.  All these can be used depending on your stack of technology and we need good designers who focus on user experience well to design it up. 

But again we have a lot of legacy applications we talked about and that requires a lot of re-engineering effort which means you have to go around hunting how to make it accessible and based on how much level of accessibility do you want to build in based on who your audience is? That kind of analysis is the main thing. 

The key thing is that if you design right upfront, social messaging is good and you get it out faster and your user community increases upfront and because it is catering to people with disabilities there is  a lot more awareness built about the product upfront and in the context of legacy applications it’s just more clean up to make your existing environment worthy of living .

You may also be aware that certain percentage of people have to be hired by a lot of organizations, government or people who work for government. People with disabilities have to be hired in their environment. How do you make it work for them? You have to go through that clean-up and there are tools mapped to technologies you have to use. 

If they are using off-the-shelf products for instance in the banking space, most of the core banking products have gone through the journey of revamping the solutions. Similarly, in retail take it segment by segment, they have re-architected it for accessibility .If you have to map it to your internal systems and at a system integration level software have to be made accessible.”

T Ashok @ash_thiru on Twitter

Summary

What does it take to do SmartQA?  Thoughtful pause, multidimensional thinking, sensitivity and awareness  and designing for robustness & testability.  A short crisp article continuing from the previous article outlining FIVE MORE thoughts, on what it takes to do SmartQA.

It is about pausing to speed up, it is about thinking multidimensionally, it is about being sensitive and aware, it is about designing for robustness and testability to doing SmartQA. 

Doing SmartQA is about visualising the act in one’s mind and taking steps to being robust and enabling rapid and easy validation.

1. Pause to speed up

Before you test, pause; ask what are you testing, for-what and for-where(environment). If you are not very clear, explore rapidly to dig in and understand from users’ POV and architecture/construction. Pause to collect your thoughts as to what issues to go after and how, then uncover exploiting using tools as needed.

After all, doing SmartQA is about enabling clarity to visualise potential issues before evaluating rapidly. Pause to speed up.

2. Multidimensional thinking

Yesterday a good friend of mine told me about his recent conversation with a senior engineering manager in a product dev org. The Sr Engr Manager, a great believer in code coverage told him that he just focuses on covering close to 100% code as the only measure to ensure quality, and as a means to implement shift-left. Absolutely right, isn’t it?  After all, ensuring all code written is at least executed once & validated is logical and necessary. 

What is/are the fallacy in this? (1) Well you have assumed that all code needed has been written (2) Well, non-functional aspects of code cannot be completely validated (3)Well, it assuming that this is what users really wanted, even if code is working flawlessly (4) Well, the number of paths to cover at the highest level of user oriented validation is just to many to cover, next to impossible! Code coverage is a necessary condition but simply not sufficient.

Doing SmartQA requires multidimensional thinking, of looking of the system from various angles both internal in terms of code, architecture and technology and external in terms of behaviour, end users, environment & usage and then making appropriate choices of what to validate later or earlier and what to prevent or detect statically.

3. Sensitivity and awareness

Being sensitive and aware to the causes of errors is useful in doing SmartQA and delivering clean code. 

1. At large, errors creep in due to these :
2. Untold expectation – Did not know they wanted these, as they did not communicate it to me.
3. Accidental omission – They missed stating it clearly to me.
4. Quiet assumptions -Filling in the gaps quietly, without confirming.
5. Incorrect implementation – Mistakes made during transformation to code.
6. Inappropriate modifications – Making fixes without fully appreciating the larger context.
7. Interesting side effects – Innocuously affecting others, without appreciating that they are coupled.
8. Deliberate abuse – Wantonly use it incorrectly to push it beyond and break it.
9. Innovative usage – Used in a very different context that I never thought about.

Appreciate that these may occur anytime, a heightened sensitivity enables us to question, analyse, clarify and validate.

Doing SmartQA is not just finding issues, but sharpening one’s senses to be able to smell these and spot them from afar or near before they hit us. It is about elevating QA to be far more valuable to business success. 

4. Designing for robustness 

Doing SmartQA is not just evaluation. It is about enabling code that is being built to be robust. To resist errors creeping in, to code-in firewalls. To ensure that I have all I need in good condition before I consume it. This implies that data (inputs) process are clean, the environment I operate in is clean and the resources I need are indeed available, and the dependencies that I have on others are indeed working well.

All I do is to protect myself. How can I handle when irritants are hurled at me? Well I have three choices :

(a) reject them and not do what I am supposed to do (b) flag them (log) and not do what I am supposed to do (c) intelligently scale down and do lesser.

The key focus is be robust, to be disaffected by inputs, configuration/settings, resources or dependent code. The act of designing for robustness makes one sensitive to potential issues that may arise and ensuring we are edged into a corner. 

5. Design for testability

Not only is it necessary to design the ‘how-to-do’, enabling ‘how-to-examine-the-doing’ is doing SmartQA. Enabling the ability to test the code easily and ascertain what went wrong to speed up debugging.

Hooking in code to enable the ability to inject inputs to stimulate and the ability to ascertain the correctness of outcomes is doing SmartQA. Putting in traces that can be logged and turned on doing test/debug and off doing production is indeed SmartQA. Embedding ‘self-test code’ is possibly the highest forms of testability. 

Ding SmartQA is about enabling the ability to validate too!

You may find the prior article on this “FIVE thoughts on doing SmartQA” interesting. Check it out!