I have an undiagnosed medical condition, I can’t leave any thread open. Have to close them, otherwise, I lose sleep. Literally.
Too many threads open at a given point of time drove me to be anxious and anxiety never feels good on anyone. I would stay awake, working on threads that are not closing. The only way I would go to sleep is that my body passes out of tiredness or my eyes start irritating. Everytime I woke up from such a body-forced-sleep, I would feel very tired because my brain would have remained active trying to solve those threads. This made me obese and put my health at risk.
When shit was about to hit the roof, I had to do something or I would just collapse. I embarked on a journey to discover myself. It led me to understand that I can only deal with a certain number of threads at any given point of time beyond which I start to break myself.
After years of practice and team building, I don’t take up too many threads today. However, the threads I pick, I do everything required to close them in a good way to move needles. That “everything” means break down the threads to small achievable goals, learn, unlearn, help people, trust people, persist, remove blockers, remove myself and partner with people to help close threads. A journey from being a hero who fights 100 bad guys at the same time to being a team member who is protected from many bad things.
I get a very high quality sleep these days. I average 50% of my sleep time into deep sleep. The days I go to sleep around 10 – 1030 PM, I get a good 7-8.5 hour sleep with a 3.5 – 4.5 hours of deep sleep. Even with a 6 hour sleep, I have about 2.5 hours to 3 hours of deep sleep. That’s because I don’t go to sleep with too many threads open. My life depends on closing threads. It always has. It always will be.
Sleep & software quality
Study on sleep deprivation on developers’ performance
Professors Davide Fucci, Giuseppe Scanniello, Simone Romano, Natalia Juristo conducted a study in December 2015 on the impact of one night of sleep deprivation on novice programmers and the impact that has on code quality and building functionally correct software. I did some research about these professors who I do not know. I was glad to discover that they seem to have done some deep study on Test First Development, Unit Testing, Impact of Unit Testing on Software Quality and Engineering Practices that Impact Software Quality.
Their paper, is a fantastic read. 20 pages of absolute bliss if you have two or three hours. I took close to five, as I am a slow reader.
Their “quasi experiment” as they call it, for reasons described in the paper led to discovering and conjectures with one night of sleep deprivation:
- Developers were 50% more likely not to fulfill the functional requirements with respect to the code produced by developers under normal sleep condition.
- Decreases developers’ engagement with the development task and hinders their ability to apply the test-first development practice. It was observed there was a difference of about 44% in the engagement with the task of developers who forewent one night of sleep and developers under normal sleep condition.
- Moreover, the results showed that sleep-deprived developers performed 54% more fixing addressing syntactic mistakes compared to developers who slept regularly.
Given the practical experience you and I have, we know that their findings are consistent to what we have seen on the projects we have worked on. We often see bugs from even skilled developers who wouldn’t usually introduce but they do due to sleep deprivation and other factors. During sprint releases we have seen developers and QA burning the midnight oil – creating a cascading lack of sleep impact. Plus, the lack of practice of unit tests is a killer combo.
Lack of sleep + lack of unit tests = More QA Jobs.
Poor quality software = more sleepless nights
2008
A decade ago, in a project, we had ~ 600 major tickets open and no engineer bandwidth to deal with it. A lot of bugs were re-opened and the chaos was exploding on our face. The Director of Engineering had an experience 8 times mine and I was a rookie Test Manager in my late 20’s.
He seemed a calm nice person but the pressure was getting to him and in one of the meeting he lost his cool. His team that includes me were put to defend why there are so many open issues not resolved. He went to his boss who said, “Sit with Pradeep and triage these”. We sifted through the data and came up with 89 bugs out of hundreds that need fixing, on priority. New feature development was paused and the fixes took priority. This helped all our lives become better. Including that of our enterprise customers who were raising support tickets left right and center.
Prior to that exercise of fixing things that matter, our night outs (working late in the night), as people call it, increased. The late night cab availing service reimbursement in the company kept increasing. It reached an all time high. We managers had to explain the charges we had approved. The company at that time felt it was better to pay late night cab charges than hire more engineers. I guess they could have invested that money to hire more engineers. Anyone’s guess.
2021
Fast forward 2021, 85% of the world’s software engineers live sprint to sprint. Most people work late nights as they get closer to the sprint release date. The open tickets and tech debt being created is pushed as backlogs. New feature requests take preceedence over improving quality. No time for unit tests because they don’t help speed. Startups that chase competition’s speed are actually chasing the competition’s scope creep speed.
Regression is the biggest unsolved problem fast moving companies are trying to address. They are bringing more people to tame the regression issue. They are doing a lot of automation that by itself requires maintenance. Jobs created as a side effect of regression don’t have a future and is full of anxiety because you close one – something else gets opened up somewhere.
Young engineers are sleeping less and late. We have taught them to take pride in it and they think they are doing great work when they have to stay late and save the project. They are wired to think that the harder they work the more they get paid. A team that sleeps less will create more regression bugs.
Preventing bugs
Preventing bugs should become the focus of every team
We say a lot of things to look cool although we don’t have an iota of motivation to follow it. One such is, “Prevention is better than cure”. The number of people who do not wear masks or wear masks incorrectly are staggeringly high. They either contract the virus or spread it.
Preventing bugs in software is similar to wearing masks. The world knows the source of bugs : at the story level and at code level. Any activity that happens outside of it is not prevention. It is detection.
A good culture brings Biz-Prod-Dev-Test together
I was a senior test engineer at a large IT services company on a project to build a Media Player app on a Windows CE smart phone (2007). The developers were rated based on how less bugs per kilo lines of code and the testers were rated the exact opposite, how many bugs found per kilo lines of code. I enjoyed finding bugs. I found a lot of them. Got a lot of appreciation for it. One day in a meeting, I realized how poorly my dev colleagues were being treated for the bugs. Their leaves were cancelled, one guy had to skip his first Diwali after marriage (which is a special one) to sit in office the next few days and fix bugs. All my pride of finding those many bugs vanished.
It is inhuman to celebrate our success at the cost of others. The same day, I decided, I won’t do my tests away from devs. I would do it with them and also they would do my tests. Gave them a battery of my test files I had created and asked them to use it as a part of dev test. Also my test ideas, heuristics and oracles that I was using to find those bugs, was all theirs.
Till that point, I had held on to my test files like they were my babies. I wouldn’t share that with anyone. I created them with so called deep learning about codecs and container formats. So, they were mine. I had investigated and used tools to create multiple formats with multiple possible error scenarios. Now, they became a part of developers. They used these files for dev-test. Their ratings went up. They prevented bugs from occuring. The client was happy and renewed the contract for the next 2 years.
We had at least 8 back to back releases to our client where no P0, P1 bugs were reported by the client or from the field. I quit that job the day the customer said, “You are a great team now”. Much before I decided to quit, I told my Senior VP who came up with the metrics of how to assess dev and test value that he was wrong and why he was wrong. He invited me to his office and said, it works for every other team except yours. I told him because we work together as one.
A culture that brings biz-prod-dev-test together is a culture that will produce healthy people producing high quality software.
20 bug prevention stories worth sharing
Moolympics is a community initiative of Moolya. We are collecting stories worth publishing. “How did you prevent bugs?” was one of our competitions several engineers in test participated and were rewarded for sharing insightful stories. We filtered about 20 stories worth publishing on how people prevented bugs.
A 50K feet summary of those 20 stories : people went deep, went close, went into the shoes of people building the business, product and code to identify something that if they had built or continued to build – would have costed them money, time and customers. Our industry needs more of this. Repeating the following for good reasons, “Most orgs today knows what needs to be done to build high quality software and also know they are not doing it”. Some need help to build practices and others are dying a slow painful death because their leaders are unwilling to change.
Closing bugs
Jira does not track and close issues, you do
They say this on their website. Right at the start. It is you who do it. The focus is on you.
Jira is two decades old. Was born as a project and issue tracking software. Atlassian has modernized it. Jira needs re-invention and am sure their Product Engineering teams eating their own dog food in Atlassian might already be thinking about this.
The focus needs to move from issue tracking to you. Even with you, it has to help you close the bugs faster and help your teams have a good sleep. The best software teams not only ship early and often but also ship quality software.
Software quality can only be achieved if bugs are prevented at large, found early and fixed effeciently. However, we all know the state of unit testing today. Everyone knows they need it but not many do it.
Since people aren’t going to build unit tests that scale, we will continue to need more than usual number of QA Engineers per product who will obviously find more bugs. This then gets to a cycle of tracking, managing, triaging and closures.
This opens too many threads in the minds of every developer, QA and product management professional to address. If there are 3 balls thrown at you between short intervals – you are likely to catch all 3. Imagine 300 balls thrown at you in short intervals and at various speeds and sizes, the ability to catch even 10% of them is near impossible. That is what is happening with these issue tracking platforms. They just record the number of the balls thrown at you. The size, speed and angle. You still have to find a way to resolve them. So, the real work is happening outside of an issue tracker.
Closing bugs is a team sport like football
If you watch any team sport like football or basketball. There are passes. Successful teams do passes really well. Closing bugs is a team sport. Pass -> Pass -> Pass -> Goal.
Football wouldn’t be fun if it was just a Pass -> Pass -> Goal. Most of the times it is Pass -> Defend -> Pass -> Steal -> Defend -> Pass -> Pass -> Shoot -> Fail -> Defend ->Pass ->Shoot ->Goal. Like this one.
Football is timeboxed, so are sprints. If the teams are trailing by points or goals, they have to make up for it in the subsequent matches. That is one way of looking at backlogs but only if they are on the border to win or qualify to the next stages. If they Software teams creating plenty of backlogs and tech-debt, they are creating poor quality software and it doesn’t matter how often they ship.
The best teams don’t find and fix too many bugs. They prevent it. Even when they occur, they have the luxury of focus on just a few threads. In our projects, we have many threads in parallel. Focus becomes a problem. This ensures the top critical ones get all the attention and the rest of the issues are center-back-logged.
The best footballers pass the ball to the place where their team members will be and not where they are. The best footballers also are fully aware while tackling where their team members are. A good product and engineering team also does something similar. They know how to report bugs for the right person to make a pass or close.
Help people close bugs
In 2019, I setup a lunch meeting with a Product Owner of a Large Health Tech company to discuss how QA Test Engineers can collaborate better with Product Teams. In that discussion, he talked about bug reports and showed me one. (I had signed the NDA with that company)
The bug report had a description with Expected API response and Actual API Response (actual response copy pasted). He said, “I have been a Dev myself and I understand this report but as a P.O. I need to know the impact of this issue to help me prioritize”. He added, “Knowing and talking about impact the bug can have on customers and our business makes a huge difference. Developers have similar cry too. Lack of tech details to help understand and fix bugs wastes everyone’s time. A poorly written bug report wastes time of the entire team.
This is not just the responsibility of people whose role is Test but everyone. Orgs who see bug tracking and closure as the responsibility of people in QA Test roles suffer poor quality software and hence sleepless nights.
Great teams don’t just ship software often, they do ship useable software, often, without adding too much tech debt to re-visit.
Semi final thoughts
Business, our profession and the work we do are tools to help improve our lives. What impacts one – impacts the other. We work for good food, good sleep and good family time and financial health. We are screwing it up when we do work in a way that comes back and kills us (makes us sleep less).
There are simple known ways to produce high quality software. Most of the engineering teams know it. It requires cultural alignment from leaders. When teams stand together and ask their leaders to bring a change, they will. A team that sleeps less, often, will produce poor quality software, irrespective of the skills they have or are learning.
This world doesn’t need fresh education on software quality. They need to put to practice what they already know.
16Prevent as many bugs as possible and have very few bugs to close. Help people to close them. Don’t leave it open. Only then you (all) can sleep well.
