New role : Product Owner’s Testing Assistant

There are enough good people on this planet now advocating unit tests, quality as a culture of the team and plenty of automated functional and integration tests written by developers and just enough front end testing that covers end to end testing and having a coach to enable testing across the team – as a culture of engineering to build quality. I do vouch for it too.

Why I vouch for it?

I have been sick and tired of being a downstream tester in the past. As a downstream tester, I was finding plenty of bugs that had escaped the source at which the bugs occurred. It was too late to fix issues. People had lost motivation to fix those bugs and had moved on to writing new dirty code.

Building multi layer automated nets help filter the bugs to minimum as we go downstream. So testers downstream would be helping us learn what we need to do, to improve our test-nets at unit and integration layers PLUS find bugs that can only be found by humans.

This is how testing should be for engineering teams. Sooner or later we are going to have approaches and tools to enable people to benefit from this approach. Start-ups and tech-focused companies are certainly leading the pack of this transformation in the world today.

Testing is not just a function of Engineering

The point that most people forget though is Testing is not just a function of Engineering. It is also a function of Business and Product Teams. However, the Engineering Team is known for hiring Testers, QA and SDET’s. So, the world associates Testing to Engineering functions much more than Business and Product.

Every company in this world is a testing company

Testing is a common terminology with Entrepreneurs and Product Teams. Testing as we all know – is a feedback loop. The quicker and more objective the better.

What do different people in different functions test?

• Entrepreneurs test their Customer Segment and Business models.
• Product Owners test their MVP, Hypothesis, Product-Market fit and Competition.
• UX Teams test their designs and flow against usage.
• Marketers test Geography and Channels before pouring more money.
• Sales teams are testing their pitch.
• Engineers test their code and integration with others code.

Engineering is a central service function to enable everyone in the company to run different kinds of tests they want to run. It requires a service mindset to be an Engineer and succeed in it.

The following list remains same across business functions.

• Test early
• Test often
• Test every change
• Test when environment has changed
• Improve sampling methods
• Automate
• Fix
• Re-test
• Retire tests that don’t add value
• Come up with new tests
• Avoid regression
• Prevent bugs

Successful Products are born out of Continuous Testing

It takes a lot of testing before the investors are ready to put in money into a product start-up. It takes a lot of testing before Product Owners find their Product-Market Fit. It takes a lot of testing for Marketers to find the right positioning. It takes a lot of testing for the Sales people get the right pitch to the right customers. It takes a lot of testing and iterations for UX teams to get the UX the way that enables people to pay without hassles. It takes a lot of testing from Engineers to get their code stability up.

It takes the entire company to get their testing right for a product to succeed with its customers.

Why only Tech Teams hire Testers and not other functions?

If every business function is really testing so much then why only tech teams have so many positions open for Testing? This is a question I have been asking for a while now. The answers came when I donned the hat of a Product Owner. I am playing a role of co-product owner for the last year and half of a tool that enables testing for everyone testing front end named as Bugasura. As the world is moving to get everyone to test – we wanted to build a tool that enables people who test front-end to reduce the pain of reporting by enabling in-app type of reporting, making it easy to explain complex front end issues to developers and other people in the organization.

Being the Product Owner of Bugasura has been an important education for me in understanding Testing that surpassed (and equally enabled) a decade of being a hands on tester.

I see testing from outside in perspective having seen testing from inside-out for over a decade. I closely work with UX, Marketing and Engineering teams who test for integration, runs automated tests and human driven tests but don’t write unit tests although they have a culture of code reviews.

I test too. I test what we build from a perspective of being a Product Owner. I observe patterns of usage from various analytics and understand what we could do better and funnel the feedback to UX and Engineering teams. I then use the knowledge I have gained on how users interact with our tool to suggest changes or find bugs that would otherwise lead to higher churn. I do test the apps we build (once a hands on tester, forever one) and report bugs. Less of “this app is crashing” but more so of discovering cases such as…

How do we find such bugs?

“I had installed Bugasura on my wife’s phone to test for compatibility a few months ago. She never opened it after that. Of course, why will she? She hadn’t enabled auto update of the app. We had at that time in Bugasura not introduced a Pro plan. There was just one Free plan. Several versions later – we introduced the pro plan. Last week she asked me for some help with her phone, noticed Bugasura and updated it to the latest version (skipping through many versions in between) and when I opened I was taken to a plan page but was locked out and couldn’t select the free plan”

This is a blocker. When someone can’t choose the free plan there can’t be a bigger blocker that this in any app. However, this issue did not exist for users who were auto updating and had moved sequentially on builds we had released.

In today’s world – these tests are run in production (and bugs are found in production too). If people click on upgrade a 100 times and 50 times they are unable to – the logs will beep a signal to slack and it alerts people and someone fixes it. However, that is at a mature stage start-up. I know how long multi billion dollar funded startups take to mature their testing and monitoring services in production. Those organizations have gone through a curve of maturity to get there.

Also, this wasn’t a backward compatibility issue because the previous build to the current build worked fine for the scenario. Now, the question is – where should these kinds of bugs be found?Let us consider this a corner case. As a Product Owner I also recognize this – not all corner case bugs are bad for the product.

Most people who test approach testing of our product like testing an app. I want to work with people who see the product we are trying to build not just the app.

Anuroop Iyengar, Founder CEO, Cogknit A.I.

Partial list of things that run in a PO’s mind when triaging bugs

• Is it even meaningful to spend time triaging this bug?
• Do we really have enough information to triage this bug?
• How often will this bug occur in production?
• How many users will be impacted by it?
• Is this a 1 in 10 million occurence bug?
• Is this consistent or intermittent?
• What emotion would the user have when encountering this bug?
• Does it impact the credibility, revenue or privacy of a customer
• What is the cost of fixing this?
• What is the opportunity cost of fixing this bug versus the other?
• What is the real tangible value of fixing this bug?
• Does this bug defeat the purpose of software or the feature?
• What can we break by trying to fix this?

In hindsight I know how many people I might have pissed during my years as a full time hands-on tester trying to sell them a bug that has no value at all 🙂

Some tests take time to design, take time to run, take time to generate a report and take time to investigate and study. This requires a deep thinking mindset.

Impact of a bug

This is what I heard from a seasoned Engineer turned Product Owner:

“Engineers can be more helpful to the Product team by also reporting the impact of a bug. As an example, I have seen an API response actual versus expected. Good but doesn’t help me as a Product Owner. I like to know what is the impact.”

Ali Mashhadi

I am about to make a statement here, ahem, “I don’t think people who work as Test Engineers will ever see the perspective of a Product Owner unless they are coached to, have built products and can see the perspective of Developers unless they have written code that made to production or unless they have spent sufficient time learning what helps developers“

Most Test Engineers pick things tacitly. They join a team – see people copy pasting expected API response versus actual API response and it becomes a way of testing.

Given that Engineers may not see PO’s perspective (because they have their own world to see and it is getting an awful lot busier to be an Engineer these days) PO’s need a Testing Assistant within their team but don’t know to ask, hire or have the bandwidth to train.

Having come from being a Test Engineer into a Product role, I think I know what does it mean to be a Product Owner’s Test Assistant. So I am attempting to write a JD here.

The missing piece between Engineering and Product

What is important for each role in the entire business spectrum are different things. That is pretty obvious.

Here are things that are on the top of a PO’s mind

• Customer
• Use cases
• Purpose
• USP
• Revenue
• Usage
• Risk
• Competition

Here is what is top on Engineer’s mind

• Understanding the story
• Constructing the logic and flow
• Implications of code changes
• Coding & Refactoring
• Testing
• Quality of code
• Not creating regression

As you can see what is on Engineer’s mind is different. Most enjoy writing code that passes the test and runs in production without breaking previously written code. Only very few engineers have a sense of awareness of what other business function needs while writing code or while imagining how to write it.

Any feedback, a bug or not, that touches the above listed aspects is of concern to the PO. That information of course is available in the bug reports and other communication that happens over slack but engineers aren’t trained to filter it out as per P.O. needs.

There are two trends happening today.

• Tech focused testing jobs are increasing exponentially
• Hence most education testers are having are focused on tech

For example, here is an excerpt of the JD for SDET at Hoststar

This is hardcore tech-focused engineering role. Most JD’s for Testers today resemble this. This is a very busy job. No ordinary. I know despite having a good team size – they keep trading off many tests and opportunities because the backlog is huge.

Hardcore engineers want to solve problems through code. Also write code that does not require too much maintenance. Also they are very good at finding problems they can solve through code. Would it interest them to find corner cases? Would it interest them to setup and run tests that take a long time. Their combination of laziness, impatience and coding skills is very important to solve a number of problems, automate things that should be and move fast.

There is a mindset, career aspiration and skills mismatch if we expect Engineers to play the role of assisting PO’s Testing Needs, unless there is a career aspiration to move into a Product.

The role of a P.O. Testing Assistant requires an objective learner. Someone who hasn’t learnt anything deep in their life won’t fit this role. So that rules out a lot of people who pretend to have testing or coding skills and have just managed to do enough to retain their jobs or get the yearly hikes. Also this is not a role for people fresh out of college. This role requires a background of having been a programmer or a tester with a product aspiration or a product person with an engineering background and aspiration.

JD for Product Owner’s Testing Assistant

Who knows the value of a role that isn’t popular or available in sufficient numbers? Who knows the value of this role. Here is my attempt at writing a detailed JD to help explain the role and the value from this role.

Find Answers

Find answers (by building dashboards) to questions that have Yes / No answers

• Have we covered the most important flows for this release?
• Do we have any issues on the critical flows that our users take?
• Have we covered the devices / browsers / environment our top 80% customers use?
• What are the top 3 issues I know about this release?

Find answers (by testing) to questions that need deep investigation like

• How is our Product performing compared to our competitor A, B and C
• What issues plague our competitors?
• What issues are plaguing our product?
• What does business want to know from us about product quality?

Being available on behalf of P.O. for

• Help people refine the definition of “Done”
• Pair test with tech teams
• Run A/B Tests
• Run User Testing
• Answer questions when tech teams are building tests
• Clarifying questions about the epic or the story
• Reviewing the usefulness of tests
• Participating in retiring tests that don’t add value to Product teams
• Coaching Tech teams on deeper Product understanding
• Appreciate prevention or detection of bugs that could have resulted in loss of value
• Advice Go-No
• Bug triage

Build Tools, Analytics and Dashboards

• Build Analytics driven test automation
• Customer usage analytics versus test coverage
  • Top used device / environment list versus test coverage device list
  • Top used flows versus tested flows
  • Heatmaps : user versus tested
• Trigger alerts when variance in test versus real usage crosses a threshold
• Bugs minus noise
• Classify product risks versus tech risks from feedback / bugs / issues

All this results in

• P.O’s time being saved and increases time available to focus on market
• P.O. getting more contextual information to make decisions
• P.O’s enabled to make better informed decisions along with the CTO
• P.O’s finding answers to questions they have in mind to improve their learning
• B.A.’s enabled by having a SPOC who can build dashboards reducing their overheads.
• Improved alignment of Tech and Product
• Deeper engagement of Product in Tech and vice versa

and thereby

• P.O. should be able to bring better revenue
• P.O. should be able to retain customers better

Preferred background

• 8+ years in the IT industry as a Programmer, SDET or Exploratory Tester with Coding skills
• Inclination and Career aspiration to a Product role
• Added bonus if served in Tech or Product Support for a while
• Added bonus if tried being an entrepreneur and built apps
• Added bonus if built tools that found usage within and outside the company
• Should have built and led teams
• Strong recruitment skills
• Leadership skills
• Should have a flair of UX
• Must understand money
• Objective problem solving skills
• Analytical thinking and visualization skills
• Be conscious of bias
• Deep learner
• Very strong people connect skills
• Communication skills that influences people
• Deep understanding of any industry
• Being very good at at least one programming / scripting language
• Analytics : GA / Crashlytics / Segment / Domo / Mixpanel
• Design : Invision / Balsamiq
• Management tools like Trello / Jira / Asana

Career Path

• P’O’s Testing Assistant to
• Lead P.O’s Testing Assistant (when built a team focused on Product Testing) to
• B.A. to
• P.O. to
• CPO

Case studies

Jokin Aspiazu Jensen

Jokin moved from being a Tester to Product Owner at Flywire. I recently had a chat with him to find out his experience and what benefit did a background of testing offer to him as a Product Owner.

His experience was interesting and his company certainly found good value in the kind of questions he was asking to the Product Teams and gave him the opportunity to be the Product Owner (who is test savvy). He is looking for new opportunities due to Covid19 situation and if you are looking at a bright person to lead your Product and or Testing. He would be bring tremendous value. He is based out of Valencia, Spain. Many people in the Testing community at MoT would vouch for him as much as I do.

Jon Bach

Jon Bach is the Master of such kind of a role. Although I wrote the title as “New Role” – Jon Bach has aced it at eBay. He joined eBay as a Lead Tester for Buyer Experience Team and found ways to demonstrate test value to Marketing Teams and today he is responsible for Marketing Tech Quality for eBay. He would be a great fit for being the P.O’s Testing Assistant and am sure he is not looking for a new role right now.

Semi final thoughts

I love the idea of semi final. Final – is conclusion. Once I conclude I can’t change. If I change then I haven’t concluded. Learning helps in not concluding. Learning helps you be in a semi final forever. Semi final is so close to final that the learning and winning spirit is very high. You are free to contribute ideas. Please message me on Linkedin.

Product mindset

Building products enables us to understand a problem as close to the reality. We go with ton load of assumptions but building a product bursts all that myths and make us get closer to reality.

When I started to build bugasura.io – I had certain assumptions of what the pain points of people are. Only when we opened our mind and mixed what users were saying to what we want to build – our product usage started to increase. Through the experience of building this I have understood testing better than what I did in the past. Most importantly I have understood the pain of people who test from front end. Not just Test Engineers pain but also of the Product Owner, UX and Developers.

Engineers are builders and hence the need for a Product mindset is much more than ever.

Keep looking at the gaps

The software industry is a fantastic industry to learn. Many engineers have found their expressions through the code they write or with the tests they run. However, most lack the big picture of how does their work fit into the big jigsaw of helping the customer succeed. This creates plenty of gaps and there are great customized career options for roles that fill the gaps that shall continue to exist.

Career guideline

The secret ingredient to finding success in a career is by understanding ourselves. Who we are and most importantly who we are not. Typically people try to pick up skills that are more in demand because they think their chances of success is directly proportional to the number of job openings. Successful people don’t do things because the world is doing it. Everyone’s path is different.

Engineers should either be super fundoo technical or super product savvy. Being neither here nor there will result in a roadkill of the career in the mid-career stage.

The role of a P.O. Testing Assistant may not be plenty in number and also may not be a role that a company is advertising for. However, if one can demonstrate value – any company would create that role.

You are likely to succeed in a role that is created for the value you bring in.

You have thoughts?

I welcome you to share your thoughts and ideas about P.O’s Testing Assistant Role with me. You can message me on LinkedIn. I shall be happy to keep updating this post with ideas that come by to benefit people who might land up here. Yet, it shall remain semi final.

14