Despite our best efforts there will be bugs in production. For bugs which impact customer perceptions of quality, it’s important that we learn from them and try to prevent similar bugs.
As a Test Manager, I would regularly review newly-reported bugs and analyse what testers could do differently to improve our processes, our skills and our approach. Recently, while working more closely with developers, I’ve revised my approach to prevent bugs earlier and therefore more efficiently.
Initial bug review process
While working as a Senior Quality Coach at Octopus Deploy, I had freedom to experiment with different approaches. This was my initial approach for bug reviews:
- Review recently raised bugs, working alone
- Categorise them by likely root cause
- Plan improvements to testing processes
- Provide recommendations to the team
Very quickly, some major flaws became obvious with this approach. You may have spotted them already.
- “… working alone” – We don’t know what we don’t know. I had major blind spots due to relying only on my own knowledge and experience.
- “… likely root cause” – i.e. possibly not the actual root cause.
- “Plan improvements …” – if the identified root cause is incorrect, how effective can these improvements be?
- “Provide recommendations …” – I’ll leave you to imagine how this went, based on all of the above.
Quality Coaching mindset
I worked with the Engineering Manager to develop a new approach to bug reviews, which was much more effective. I took off my Test Manager hat and put my Coaching hat back on. This new approach could be summarised as “Ask, don’t tell”. Our goal remained the same: Understand the root cause, how the bug was detected, how can we prevent similar bugs and/or detect them sooner.
Monthly I would meet with each developer for 30 minutes, and together we’d review ~3 bugs they’d fixed recently. This was the base set of questions we’d cover for each bug:
- How difficult/easy was it to investigate and discover the cause of the issue?
- Could we have prevented this issue from reaching main branch with a safety net*?
- Would a safety net for this issue cost more than it’s worth?
- Can we detect this type of issue before our customers do?
- Did you find related issues while fixing this one? If so, were they known issues?
Lessons learned
There’s no need to review every issue, or you’ll end up with a long list of improvements and no time to implement them all. In our sessions the developers selected the bugs we’d discuss (before or during the meeting) and I found that this made the session more engaging for them and therefore more productive overall.
Our bug review meetings were equally about coaching and learning for me. From the coaching side, I could prompt developers to consider security, performance, scale, and other quality aspects of their code. At the same time I gained a deeper understanding of specific features and how they’re implemented. For example, on an issue where I would normally have suggested additional performance testing, I learned about static analysis checks which could prevent the issue – much faster and cheaper!
After the first session or two, this “prevention and detection” quality mindset becomes an automatic part of bug fix work. Future sessions with the same developers felt more like a reminder or prompt to prioritise quality during the normal course of their work. The sessions also provided an opportunity to share any new progress, concerns or learnings regarding quality methods.
With experienced developers who are already quality-focused, these meetings might not yield much benefit and could start to feel like a hollow ceremony. Nobody enjoys a meeting that feels like a checkbox and doesn’t deliver tangible value. If you find that your senior developers already know exactly how to prevent similar issues from occurring, they may instead appreciate support in advocating to prioritise that work, or in preparing training for others in the company. You could cancel future sessions, stick with them as a way for you to learn more, and/or meet less often.
If you have multiple coaching sessions where you both agree that “xyz” needs to be done, and no progress is being made in that area, it has a negative effect on the energy and motivation for future bug review sessions. It may be time to switch gears, and focus your efforts on gathering leadership support to prioritise getting “xyz” done.
There are pros and cons of when to review the root cause of an issue. For high severity issues where customers are experiencing loss of critical functionality, of course we fix them urgently and discover the root cause as soon as possible. With that exception, reviewing bugs monthly after they’ve been resolved was the process I found most useful.
*A safety net is a test or process which will catch a bug before it impacts customers. Eg, unit test, dogfooding, user testing, etc.
This is the last post in a short series of posts on quality coaching.
Is it good enough yet? 