One of our clients is a multi-billion dollar organization who caters to millions of customers every month through their online systems and offline stuff. For one of their legacy system, we are helping them build a regression testing solution.
It wasn’t just a legacy system but an inherited one. Only the code could be inherited and not any details of tests done. So, people are still trying to figure out what works how. Even if they have cracked it, they need good tests to learn if they need to learn more or different things. Our testers are good at working when there is hardly any documentation because our ideas are diversified.
Our brain has a different piece of code running in it from what we see in the majority.
Here’s how we have seen people think:
if ( document ne ”available” ){
print “cant test well without documents”;
}
else {
@follow_the_usual_process;
@convert_document_available_to_test_cases;
}
Instead this is a code snippet of the kind of program that runs in our mind:
if ( document ne “available” ) {
print “That’s OK”;@talk_to_developer;
@talk_to_users;@read_code;
@assess_mission;
@ask_questions;
@track_status;
@track_mission_versus_whats_achieved;
@prepare_your_own_notes;
@read_claims;
@compare_against_similar_products;
@test_learning_against_biases;
@use_play;
@assume_thats_how_it_works_and_test_for_assumptions;
@build_mind_map;
@test_for_testability;
…;
…;
@track_status
@track_mission_versus_whats_achieved
…;
}
We didn’t start our careers with the above program in our head but we learnt and practiced to re-wire our brain to get the above code running. Rapid Software Testing has played a vital role in helping us not be intimidated by things that appear to block the progress to be made.
At this moment if you are thinking that we are trying to tell, “Oh, we are so good you see”, hold on.
The trap we fell into & the trap we didn’t want our client to fall into
For the data intensive part of tests, we created meta data and fed them to various tools to generate all pairs and presented the same to our client. They were like, “What, so many?” and with pride we replied, “Yeah”. To our surprise, they responded, “I think we should cut short this to a few hundreds”.
That wasn’t what we expected. Our learning about test coverage suggests that each test is different no matter how similar they may look and chopping off tests just because they are huge in number isn’t a good way to think about good testing. We tried helping our client understand that but we failed. We then presented a pairwise testing approach from TestersDesk and then shifted to Hexawise that offered tons of flexibility ( Hexawise rocks). The moment we showed the few hundred tests our client was like, “Yeah, this is what we wanted”.
It was win for them but not for us. Pairwise testing is important but not when that’s the only thing done. We then started to have conversations of All Pairs versus Pairwise. We failed in all our attempts to help our client see the picture we were seeing. ( Yeah, we failed )
The reason we failed was because our conversation lacked life in it or we were completely on the other extreme. It is easy for anyone at that moment to move away from the goal in hand to try winning the argument. I personally could see that we were on the verge of losing hope that we could help our client see the picture.
Rapid recovery from traps
Thankfully, our practice to ask ourselves, “Are there lessons that we have learnt that we aren’t bringing forward?”. That’s when we started to see some hope. We learnt that we hadn’t been presenting our client with evidence of the coverage missing if pairwise is our only approach.
Rapid testing isn’t a bible but a set of heuristics that are very important for a tester who wants to shine amidst tough situations or do much better in favoring situations. When we were talking about test coverage, being of good value to our client, helping them solve their testing problems… we were drawing those lessons from RST. However, we fell into a trap of losing track of the mission we had in hand and how to get there for a few brief moments ( about 2 weeks, actually )
We fell into a trap as we failed to bring in other set of heuristics from RST.
If I you think I am telling; to be successful one should use all heuristics of RST then it is clear you don’t understand what a heuristic is 😉Its simple, a heuristic is an idea that usually works and could fail too. The choice of heuristics depends on the context. The application of heuristics depends on the skills of a tester.
That’s when we started to gather more evidence to bring life into our arguments and why our client needs more than just pairwise. After a brief chat with Justin Hunter – The Hexawise Guy, I was glad to discover that he too was advocating the same thing – pairwise only approach is bad.
However, here is an interesting thing. After having grown wiser over the years, I personally ensured that testers at Moolya don’t fall into the trap of thinking, “Oh this client doesn’t understand testing”.
One other thing about RST that was helpful was quick recovery from traps. When James Bach, Michael Bolton and other trainers of RST that may even include me, train people – we try helping testers get into a trap and see how they recover. We help them build the skills to recover from traps. RST gets the reality of testers well. No matter how smart a team of testers are, its hard to avoid all traps. So RST focuses on helping the smart tester get more smart in teaching how to recover from traps. Having done at least enough exercises from RST, continuing to practice it, we ended up recovering well.
Reading & Understanding Code without being blocked
We wanted to run a code coverage tool and then run the pairwise tests to determine what could be missing but the legendary legacy code wasn’t built to facilitate that quickly. We didnt have code view permissions. For us, developers add to our strength and it so happened that they allowed us to view code, copy a small function to our machine.
While being trained on Rapid Testing, figuring out work around for things that appear to cause a road block is a routine. Our brain thankfully was so used to it that we got developers with read and write permissions of the code to help us view the code. So, when you don’t have the eye for looking into something, get someone who has it and can guide you.
We started to read code and tried understanding it. That gave us a big breakthrough. BTW, our process to understand code was a nice journey. We saw that the code didn’t have enough code comments for us to learn or make good inferences, so we wrote code comments for it and sent it to developers requesting them to correct us.
That’s the way we pinned down on things. We found:
- Code paths and functions that are critical but aren’t exercised by Pairwise
- Error messages and strings that wont show up with Pairwise tests.
- How pairwise tests would touch and go important functions without exercising them enough?
- How regressing a code change would be incomplete with just the pairwise tests?
Building our case and presenting it to our client
Added to that we did other bits of investigation:
Thankfully, we had view access to bugs logged in the past, those that escaped, those that re-opened. We started marking bugs that pairwise tests would help them catch and those that would be missed.
We went to another level of investigation on those that pairwise would miss. We were paying attention to the pattern of problems that are missed and identified scenarios that could help in catching those, which were out of the scope of Pairwise.
We then made a presentation of it. Our tone was so mature when we spoke to our client this time. We had solid facts, solid figures and solid evidence. We are proud to tell you that our client was able to see the picture we wanted them to. We have been asked to submit a revised plan for strengthening the regression test suite and we are working on it right now.
Building multi layer safety nets & thanks to Google Test Automation Conference ( GTAC 2010 ) we attended in Hyderabad last year
The progress we have made so far is encouraging. We heard quite a few testers (who write code to test) inGTAC talking about safety nets and multi layer nets. At that time we just were patiently listening to all of it sipping the unlimited supply of beer Google offered us. While the beer went into the body and came out the ideas went into the mind and stayed.
We appeared to pull the concept of safety and feedback nets and layered approach to pairwise. Hexawise does that for us. We could choose 2 way, 3 way, 4 way, mixed 3 way, mixed 4 way layers.
The idea is to help developers use these multi layer nets even before a build is pushed into the test environment. These act as acceptance tests for the test team to start looking into a build.
After running 2 way, 3 way the mixed 3 way and mixed 4 way offers us an quicker way of investigating what component when interacting with a specific component is appearing to cause a problem.
So, we are assuming that our client has got more value out of the pairwise approach and will be getting more value out of other approaches we are going to use to test. Assuming because, we are mid way. There’s a long way to go and the most important one now that we have crossed half way through it.
This is awesome and so much fun.
On a side note: There are couple of announcements coming over the next week, so be sure you are subscribed to our feeds, Twitter, Facebook Page and oh if there is Google+ pages, we will be there soon.
Unlike other Indian IT companies, we are going to bring all that we do to the outer world. We hope you enjoyed reading what is going on inside Moolya and we look forward to having your support.
0