A mashup of three articles published as part of SmartQA digest over the last few months that outlines mindsets needed to brilliant code, tips to produce clean code and habits to speed up evaluation
Here is a mashup of three articles published as part of SmartQA digest over the last few months that outlines mindsets needed to brilliant code, tips to produce clean code and habits to speed up evaluation.
10 Simple Tips to Clean Code has in detail, simple practices that can ensure that code developed is constantly cleansed. In current times where code is churned out at a rapid pace, it makes great business sense to contain the entropy continually.
Automation is a key enabler for rapid QA, the limiting function to speed is one’s skills. And this is where good habits come in. 10 Habits to Help You Speed Up Testing outlines these in detail.
Doing SmartQA is about great mental clarity of visualising what is intended, what is present, what-may-be-missing that could-be added to enhance the experience. The intent to seek this clarity result in good questions that help us understand better and therefore test well is the objective of this article. Two tools for the mind “Landscaper” and “Deep diver” that can help are outlined here.
Prevention occurs due to good understanding. Detection occurs due to good understanding. Understanding of what is needed, what is stated and what is implemented.
Doing SmartQA is about great mental clarity of visualising what is intended, what is present, what-may-be-missing that could-be added to enhance the experience. The intent to seek this clarity is what one drives to question well, build better, prevent and detect issues.
The act of testing is really discovering what-should-be-there but-not there, what-is-there but not correct, what-should-be-there but should-not-be-there. Finally it is about understanding the impact of something that been changed, be it in the system or outside the system.
The key to this is understanding the product/application/solution. Understanding from different points of view- end users, construction, technology, environment, development & deployment.
Smart Understanding is about scouring the ‘landscape’ to understand overall context and the static structure of how it is built and then ‘deep-diving’ to understand the intended dynamic behavior. Landscape and Deep dive are great tools for the mind to explore the system rapidly to do SmartQA. The associated picture illustrates these two thinking tools well.
Tool#1 Landscaper
Given an entity to test, be it a small component or a big business flow or the entire system, the first thing to do is examine it well by performing a ‘landscape’ .
Start with understanding who the end users are, then identify the entities (the system offerings), the connect to who uses them and how much and then go on to figure out as to what they may expect from these.
Now switch to a deployment view to understand how is deployed and other systems it is linked with.
Now from the construction/development point of view understand what is built new (via fresh code or glue code that integrates components) and what is being modified.
Continuing to look from the structural view, understand how these entities may be coupled, to appreciate the interactions and what may be affected due to the new/modified entities.
Go a little deeper to understand the system is is architected and technology stack(s) used to implement this.
What we are doing is doing a tour from different points of view and attempting to understand the ‘whole’. During this process many questions arise which enables better clarity of problem.
Tool#2 Deep diver Having a good holistic picture of the system under test, it is only natural to to dive deeper to understand in detail an entity. Deep diver is the second tool that helps you do this.
First understand the various inputs to this entity. What are they, where do they come from, what is their format and spec, rates and volumes too? Are there any interesting values ?
Next, understand what the various outputs are and may be. What are the normal outputs and what are those in situations of error? Do check if ‘all’’ possible outputs have been indeed been considered.
Finally it is time to understand the intended behavior by discovering conditions that transform the inputs to intended outputs. Note that some of the behaviour conditions could be based on system state and not based on inputs only.
Smart understanding is key to doing less, doing well and accomplishing more. This is a seriously mental activity and doing it well has great paybacks! The two mental tools “Landscaper & Deep diver” enable a logical approach to decomposing the system(i.e. problem) well so that you have great clarity on what, how-to-do and how-much-to-be-done.
This article outlines twenty approaches to smart test design based on seven views of user, logic/analysis, construction, test, experience, operational and evolution.
Let us talk a bit of test design now. We focus a lot of execution, and therefore the ability to cover more. The focus has veered to how frequently we are able to execute the tests and therefore on automation. Let’s step back and ask to what the objective was. Well it was to primarily deliver clean code. So we need to have a deeper sensitivity to the quality of tests. This is where design becomes important.
So what is a smart way to design to come with good scenarios, ideally few that can uncover issues that matter most.
Smart Test Design is about looking at the system from multiple views to:
I want | expect | would-like behaviours to satisfy needs that are implemented well and comprehensively covered to help me do well on my environments with no side effects.
Then decide what you want to prevent, statically detect or test, be it via human or a machine. Focus on intent and then the activity.
Let me list down TWENTY approaches to smart test design looking at the system from SEVEN views.
1. User view based
1.1 Use the requirement specification to design. 1.2 See actual users of how they work and then use this to design. 1.3 With users doing experience sessions with system and use this information to design.
2. Analytical view based
2.1 Use software test techniques (black or white) on spec/structure to design. 2.2 Construct behavior models and use this to design. 2.3 If the system needs to comply with a standard, use the standards information to design.
3. Construction view based
3.1 Use the code properties like lines/conditions/path to design. 3.2 Exploit your deep understand of technology to understand potential mechanisms/flaws and use this design. 3.2 Understand how the system has been architected, composed & integrated to design.
4. Test view based
4.1 Hypothesise potential faults probable given your understand of usage, structure, architecture, environment, conditions to design. 4.2 Use potential error return codes, exceptions, deliberate bad inputs, violations of system states to design. 4.3 Identify potential end failures and failure modes and use this to design. 4.4 Explore the system to understand its behaviour in various contexts and use this to design . 4.5 Probe the system with a series of questions (say what-if) and use this to design.
5. Experience-view
5.1 Use the past history of issues encountered with various customers and design. 5.2 Apply the learning of various situations or deeper knowledge to come u with fault patterns and use this to design.
6. Operational-view
6.1 Use the understanding of actual business flows, usage profiles of features on various environments to design. 6.2 Identify various deployment configurations and use this information to design.
7. Evolution-based
7.1 Use the information of what code has been changed, to understand how these may propagate and may cause issues to design. 7.2 Use the information of changes in the environment to understand their impact on the system and design
Smart Design is about:
I want | expect | would-like behaviours to satisfy needs that are implemented well and comprehensively covered to help me do well on my environments with no side effects.
What does it take to do SmartQA? Thoughtful pause, multidimensional thinking, sensitivity and awareness and designing for robustness & testability. A short crisp article continuing from the previous article outlining FIVE MORE thoughts, on what it takes to do SmartQA.
It is about pausing to speed up, it is about thinking multidimensionally, it is about being sensitive and aware, it is about designing for robustness and testability to doing SmartQA.
Doing SmartQA is about visualising the act in one’s mind and taking steps to being robust and enabling rapid and easy validation.
1. Pause to speed up
Before you test, pause; ask what are you testing, for-what and for-where(environment). If you are not very clear, explore rapidly to dig in and understand from users’ POV and architecture/construction. Pause to collect your thoughts as to what issues to go after and how, then uncover exploiting using tools as needed.
After all, doing SmartQA is about enabling clarity to visualise potential issues before evaluating rapidly. Pause to speed up.
2. Multidimensional thinking
Yesterday a good friend of mine told me about his recent conversation with a senior engineering manager in a product dev org. The Sr Engr Manager, a great believer in code coverage told him that he just focuses on covering close to 100% code as the only measure to ensure quality, and as a means to implement shift-left. Absolutely right, isn’t it? After all, ensuring all code written is at least executed once & validated is logical and necessary.
What is/are the fallacy in this? (1) Well you have assumed that all code needed has been written (2) Well, non-functional aspects of code cannot be completely validated (3)Well, it assuming that this is what users really wanted, even if code is working flawlessly (4) Well, the number of paths to cover at the highest level of user oriented validation is just to many to cover, next to impossible! Code coverage is a necessary condition but simply not sufficient.
Doing SmartQA requires multidimensional thinking, of looking of the system from various angles both internal in terms of code, architecture and technology and external in terms of behaviour, end users, environment & usage and then making appropriate choices of what to validate later or earlier and what to prevent or detect statically.
3. Sensitivity and awareness
Being sensitive and aware to the causes of errors is useful in doing SmartQA and delivering clean code.
At large, errors creep in due to these :
Untold expectation – Did not know they wanted these, as they did not communicate it to me.
Accidental omission – They missed stating it clearly to me.
Quiet assumptions -Filling in the gaps quietly, without confirming.
Incorrect implementation – Mistakes made during transformation to code.
Inappropriate modifications – Making fixes without fully appreciating the larger context.
Interesting side effects – Innocuously affecting others, without appreciating that they are coupled.
Deliberate abuse – Wantonly use it incorrectly to push it beyond and break it.
Innovative usage – Used in a very different context that I never thought about.
Appreciate that these may occur anytime, a heightened sensitivity enables us to question, analyse, clarify and validate.
Doing SmartQA is not just finding issues, but sharpening one’s senses to be able to smell these and spot them from afar or near before they hit us. It is about elevating QA to be far more valuable to business success.
4. Designing for robustness
Doing SmartQA is not just evaluation. It is about enabling code that is being built to be robust. To resist errors creeping in, to code-in firewalls. To ensure that I have all I need in good condition before I consume it. This implies that data (inputs) process are clean, the environment I operate in is clean and the resources I need are indeed available, and the dependencies that I have on others are indeed working well.
All I do is to protect myself. How can I handle when irritants are hurled at me? Well I have three choices :
(a) reject them and not do what I am supposed to do (b) flag them (log) and not do what I am supposed to do (c) intelligently scale down and do lesser.
The key focus is be robust, to be disaffected by inputs, configuration/settings, resources or dependent code. The act of designing for robustness makes one sensitive to potential issues that may arise and ensuring we are edged into a corner.
5. Design for testability
Not only is it necessary to design the ‘how-to-do’, enabling ‘how-to-examine-the-doing’ is doing SmartQA. Enabling the ability to test the code easily and ascertain what went wrong to speed up debugging.
Hooking in code to enable the ability to inject inputs to stimulate and the ability to ascertain the correctness of outcomes is doing SmartQA. Putting in traces that can be logged and turned on doing test/debug and off doing production is indeed SmartQA. Embedding ‘self-test code’ is possibly the highest forms of testability.
Ding SmartQA is about enabling the ability to validate too!
What does it take to do SmartQA? How can I do less and accomplish more? What parts of this are human-powered & machine-assisted? A short crisp article outlining some thoughts, five for now on what it takes to do SmartQA.
It takes a brilliant mindset, intelligent exploration, diligent evaluation, keen observational skills, tech savviness and continual adjustment. It is about being logical yet be creative, it is about being disciplined yet be random, it is about exploring the breadth and depth, it is about understanding deeply and also finding blind spots about being bundled by time but be unlimited/unbounded with the possibilities. Doing SmartQA is about doing mindfully, in a state of brilliant balance.
#1 What does it take to do SmartQA?
A deductive ability of a mathematician, creativity of an artist, mind of an engineer, value perception of a businessman, technical savviness, empathy, doggedness and nimbleness, all finely honed to do less and accomplish more.
#2 Humans & Machines : Doctors & diagnosis
In today’s medicine, we know that machines play a huge part in diagnostics and treatment. They help us see internals more clearly, enable us to get to the hard-reach parts, perform rapid tests to analyse problems , monitor tirelessly to help us correct our actions. So is the doctor’s role redundant? Ouch no! The skill of the doctor in diagnosis and treatment be it via medicine or surgery is far more required now in the complex world of disease, business and law. To assist in this ever increasing complexity, machines are becoming integral for the job.
Much like this is software testing, the act of diagnosing of software for issues. Tools/automation are integral to testing that a skilled test/QA/SW engineer uses. It is about “doing SmartQA” which is a brilliant combination of “human powered and machine assisted” . The WHAT to-do is human while HOW-to-do is when’re machine helps.
Doing SmartQA is about intelligent/smart WHAT-to-test/WHAT-to-test-for/WHERE-to-test-on with smart enablement of HOW-to-test using machine/tools. It ain’t automated testing or manual testing and getting rid of the latter or machines find issues on itself.
#3 On Minimalism
I have always practiced minimalism, of doing least work with superior outcomes. Have never been a fan of more tests and therefore needing to tools to accomplish these in the context of software. Of course I exploit tools to brilliant work. Let us apply this to ‘doing SmartQA’.
We talk about left shifting, of TDD, of wanting to find issues earlier. We emphasise units tests and automating these. Are we shifting the objective to doing more unit tests? The purpose was to produce ‘cleaner units’, implying heightened sensitivity to issues via TDD and writing good code in the first place, of adopting cheaper static means to uncover these may be missed before resorting costlier automated unit tests.
Doing SmartQA is really about ’not doing’, well doing minimally really. It is not doing more and therefore needing to adopt tools. So heighten your sensitivity, build good code habits, use mental aids like smart checklist and of course exploit software tools to do the heavy lifting of tests. After all, don’t we all want to adopt wellness rather than expend effort to diagnose potential illness? And I bet you want an doctor to check out before they ‘outsource’ the finding to machines.
#4 Brilliant engineering
Is testing a mere act of uncovering bugs? I think not. It is really a mindset to clarify a thought. When we develop something, in this case code, a smart testing mindset enables us to step outside of being the producer into
the shoes of end users, empathise, see their point of view, appreciate what their environment looks like and understand what all can go wrong so that we produce clean code. At the worst case, put hooks inside code to give us more information as to how code is being buffeted, so that we may examine later and refine the code.
Doing SmartQA is not just about finding bugs but getting into the mindset of ‘Brilliant engineering’.
#5 See better, cover more, test less
A good ‘world view’ enables us to ensure great coverage in testing. Is coverage limited to execution only or does it allow us to see better?
Coverage is about enabling us to see better from all angles. It is not about merely ascertaining if test(s) could-be/are effective. Enabling a viewpoint from USERS, ATTRIBUTES, ENVIRONMENT, CODE, ENTITIES allows us to see from multiple angles, sensitising us to deliver brilliant code with less testing/validation. Of course it does help us significantly judge the quality of test cases and testing also.
Viewpoints from USERS, ATTRIBUTES, ENVIRONMENT, CODE, ENTITIES
Doing SmartQA is more than just doing, it is about significant enablement to see better, heighten sensitivity and accomplish more.
You may find this article “50 Tips to SmartQA” interesting. Check it out!
“A typical accident takes seven consecutive errors” states Malcolm Gladwell, this notion is reflected in Mark Buchanan’s book “Ubiquity” too. This article dwells upon ‘How do you ensure that potential critical failures lurking in systems that have matured can still be uncovered?’
—
“A typical accident takes seven consecutive errors” quoted Malcolm Gladwell in his book “The Outliers”. As always Malcolm’s books are a fascinating read. In the chapter on “The theory of plane crashes”, he analyses the airplane disasters and states it is a series of small errors that results in a catastrophe. ” Plane crashes are much more likely to be a result of an accumulation of minor difficulties and seemingly trivial malfunctions” says Gladwell. The other example he quotes is the famous accident – “Three Mile Island” (nuclear station disaster in 1979).
It came near meltdown, the result of seven consecutive errors – (1) blockage in a giant water filter causes (2)moisture to leak into plant’s air system (3) inadvertently trips two valves (4) shuts down flow of cold water into generator (5) backup system’s cooling valves are closed – a human mistake (6) indicator in the control room showing that they are closed is blocked by a repair tag (7) another backup system, a relief valve was not working.
This notion is reflected in the book “Ubiquity” by Mark Buchanan too. He states that systems have a natural tendency to organise themselves into what is called the “critical state” in what Buchanan states as the “knife-edge of stability”. When the system reaches the “critical state”, all it takes is a small nudge to create a catastrophe.
Now as a person interested in breaking software and uncovering defects, I am curious to understand how I can test better. How do you ensure that potential critical failures lurking in systems that have matured can still be uncovered?
Let us look at what we do- We stimulate the system with inputs (correct & erroneous) so that we can irritate latent faults so that they may propagate resulting in failure. When the system is “young”, the test & test cases we come up are focused on uncovering specific (singular) faults. i.e a set of inputs that can irritate singular faults and yield possibly critical failures. This is possible because the “young system” is not yet resilient and therefore even a singular fault bumps it up! We then think that our test cases (i.e. combinations of inputs) are powerful/effective. But these test cases do not yield defects later as the system becomes resilient to singular faults.
As the system matures we need to sharpen the test cases to irritate a set of potential faults that can create a domino effect to yield critical failures. Creating test cases to uncover singular faults in a mature system may not useful. It is necessary that test cases be at a higher level of system validation (i.e have long flows) and have the power to irritate a set of faults.
Should we resort to uncovering critical failures only via testing? By creating test cases at higher levels that have the power to uncover multiple types of faults? Not necessarily. We can apply this thought process at the earlier stages of design/code too. Using the notion of sequence of potential errors and understanding what can happen.
If your drive in India you know what I mean … the potential accident due to a dog chasing a cow, which is charging into the guy driving the motorbike, who is talking on the cell phone, driving on the wrong side of road, encounters a “speed bump” , and screech *@^%… You avoid him if you are a defensive driver. Alas we do not always apply the same defensive logic to other disciplines like software engineering commonly enough…
TEN tips for a developer to enable delivery of brilliant code and TWELVE tips to become a modern smart tester is what this article is about. Curated from two earlier articles that I wrote.
What are my TEN tips for dev to deliver brilliant code? Here it is visualised as mind map!
Minmap of TEN tips for a dev to deliver brilliant code
This article is curated from SIX articles as a quick primer on AI. Starting with a glossary on AI, it delves into tacit knowledge as codified via ML and understanding the difference between ML & AI. A quick peek into deep learning and challenges of explaining the patterns ending with a interesting piece AI written by an AI program.
Glossary of terminology in AI
Here is an easy article “A Simple, yet Technical Glossary of Terminology in AI” that provides simple glossary of terms, abbreviations, and concepts related to the field and sub-fields of artificial intelligence. Based on technical definitions by pioneers and leaders in AI.
ML & Polanyi’s paradox
“Explicit knowledge is formal, codified, and can be readily explained to people and captured in a computer program. But tacit knowledge, a concept first introduced in the 1950s by scientist and philosopher Michael Polanyi, is the kind of knowledge we’re often not aware we have, and is therefore difficult to transfer to another person, let alone capture in a computer program. Machine learning has enabled AI to get around one of its biggest obstacles, the so-called Polanyi’s paradox.” Read more in this article ”What Machine Learning Can and Cannot Do“
ML & AI – The difference (1)
There’s much confusion surrounding AI and ML. Some people refer to AI and ML as synonyms and use them interchangeably, while others use them as separate, parallel technologies. In many cases, the people speaking and writing about the technology don’t know the difference between AI and ML. In others, they intentionally ignore those differences to create hype and excitement for marketing and sales purposes.” This article “Why the difference between AI and machine learning matters” attempts to disambiguate the jargon and myths surrounding AI.
ML & AI – The difference (2)
“Unfortunately, some tech organizations are deceiving customers by proclaiming using AI on their technologies while not being clear about their products’ limits. There’s still a lot of confusion within the public and the media regarding what truly is artificial intelligence, and what truly is machine learning. Often the terms are being used as synonyms, in other cases, these are being used as discrete, parallel advancements, while others are taking advantage of the trend to create hype and excitement, as to increase sales and revenue.” says Roberto Irliondo in his article “Machine Learning vs. AI, Important Differences Between Them”.
Explaining how patterns are connected
Deep learning is good at finding patterns in reams of data, but can’t explain how they’re connected. Turing Award winner Yoshua Bengio wants to change that, read about this is this article “An AI Pioneer Wants His Algorithms to Understand the ‘Why’“
In this article I have tried picture(ise) the landscape of the plethora of tools for testing software which has moved away from just testing to build-test-deploy in a continuous manner. Keeping the interesting visual I have listed the FIFTEEN broad categories of tools that make up the modern digital testing landscape.
Given the plethora of front ends for digital applications of today, from PCs to tablets and mobile phones wth varied form factors, OS and browsers, with varying connection speeds, sometimes uncertain, the integration with many external systems via services with demands on non-functional attributes and the frequent nature of releases has made the challenge of automation and keeping in sync harder.
In this article, I have attempted to picture(ise) the landscape of test tooling with the entity-under-test(EUT) at centre (note that an EUT may be a small component, a subsystem or the complete system) with multiple ways to access it via API, Message/Service or UI on the left, evaluation by various test types at the top, the EUT enclosed in a deployment environment that may be an container/system with various ‘platform choices’. Keeping this as the base I have attempted to enumerate the various activities related to evaluation as inject/stimulate, observe/measure, validate/oracle, log/record and generate mocks as necessary to test for functionality or other non-functional attributes in the larger context of test design, automate, build and deploy on a variety of platforms as necessary.
Test Tools Landscape – 15 Categories of Tools
Using the above picture we have the FIFTEEN categories of tools with some example tools as a table below.
#
Category
Details
Example tools
1
inject/stimulate
Enabling accessing the EUT via API, Service/Message or via UI
xUnit, SoapUI, Postman, Selenium
2
observe/measure
Enabling run time aspects of EUT
Coverage, Resource leaks
3
validate/oracle
Enabling assessment of pass/fail
File comparators, Asserts
4
log/record
Enabling logging data, test information
log utilities, test execution recorders
5
mocks
Provide stubs for yet-to-be developed code
Mockito, Wiremock
6
non-functional test tools
Enable assessment of non-functional attributes
JMeter, SonarCube
7
platforms
Enable testing on different mobile devices with different browsers
pCloudy
8
virtualisation/deployment
Enable visualisation and deployment of code
Jenkins, Tricentis TOSCA
9
mocks/simulators
Enable simulating or mocking large systems
Payment gateway simulators
10
test design
Enable design of test case via specification based testing, BDD
