RSS

Author Archives: Kim Engel

About Kim Engel

Software Quality Specialist

Quality Coaching: Preventing production issues

Despite our best efforts there will be bugs in production. For bugs which impact customer perceptions of quality, it’s important that we learn from them and try to prevent similar bugs.

As a Test Manager, I would regularly review newly-reported bugs and analyse what testers could do differently to improve our processes, our skills and our approach. Recently, while working more closely with developers, I’ve revised my approach to prevent bugs earlier and therefore more efficiently.

Initial bug review process 

While working as a Senior Quality Coach at Octopus Deploy, I had freedom to experiment with different approaches. This was my initial approach for bug reviews:

  1. Review recently raised bugs, working alone
  2. Categorise them by likely root cause
  3. Plan improvements to testing processes
  4. Provide recommendations to the team

Very quickly, some major flaws became obvious with this approach. You may have spotted them already.

  1. “… working alone” – We don’t know what we don’t know. I had major blind spots due to relying only on my own knowledge and experience.
  2. “… likely root cause” – i.e. possibly not the actual root cause.
  3. “Plan improvements …” – if the identified root cause is incorrect, how effective can these improvements be?
  4. “Provide recommendations …” – I’ll leave you to imagine how this went, based on all of the above.

Quality Coaching mindset

I worked with the Engineering Manager to develop a new approach to bug reviews, which was much more effective. I took off my Test Manager hat and put my Coaching hat back on. This new approach could be summarised as “Ask, don’t tell”. Our goal remained the same: Understand the root cause, how the bug was detected, how can we prevent similar bugs and/or detect them sooner.

Monthly I would meet with each developer for 30 minutes, and together we’d review ~3 bugs they’d fixed recently. This was the base set of questions we’d cover for each bug:

  • How difficult/easy was it to investigate and discover the cause of the issue? 
  • Could we have prevented this issue from reaching main branch with a safety net*? 
  • Would a safety net for this issue cost more than it’s worth? 
  • Can we detect this type of issue before our customers do? 
  • Did you find related issues while fixing this one? If so, were they known issues? 

Lessons learned

There’s no need to review every issue, or you’ll end up with a long list of improvements and no time to implement them all. In our sessions the developers selected the bugs we’d discuss (before or during the meeting) and I found that this made the session more engaging for them and therefore more productive overall.

Our bug review meetings were equally about coaching and learning for me. From the coaching side, I could prompt developers to consider security, performance, scale, and other quality aspects of their code. At the same time I gained a deeper understanding of specific features and how they’re implemented. For example, on an issue where I would normally have suggested additional performance testing, I learned about static analysis checks which could prevent the issue – much faster and cheaper!

After the first session or two, this “prevention and detection” quality mindset becomes an automatic part of bug fix work. Future sessions with the same developers felt more like a reminder or prompt to prioritise quality during the normal course of their work. The sessions also provided an opportunity to share any new progress, concerns or learnings regarding quality methods.

With experienced developers who are already quality-focused, these meetings might not yield much benefit and could start to feel like a hollow ceremony. Nobody enjoys a meeting that feels like a checkbox and doesn’t deliver tangible value. If you find that your senior developers already know exactly how to prevent similar issues from occurring, they may instead appreciate support in advocating to prioritise that work, or in preparing training for others in the company. You could cancel future sessions, stick with them as a way for you to learn more, and/or meet less often.

If you have multiple coaching sessions where you both agree that “xyz” needs to be done, and no progress is being made in that area, it has a negative effect on the energy and motivation for future bug review sessions. It may be time to switch gears, and focus your efforts on gathering leadership support to prioritise getting “xyz” done.

There are pros and cons of when to review the root cause of an issue. For high severity issues where customers are experiencing loss of critical functionality, of course we fix them urgently and discover the root cause as soon as possible. With that exception, reviewing bugs monthly after they’ve been resolved was the process I found most useful.

*A safety net is a test or process which will catch a bug before it impacts customers. Eg, unit test, dogfooding, user testing, etc.

This is the last post in a short series of posts on quality coaching.

 
Leave a comment

Posted by on September 4, 2021 in Coaching, Quality, Software Testing

 

Tags: , ,

New Feature Development: Working with a Quality Coach

In the previous post I’ve listed some responsibilities of the Quality Coach role. One activity I’ve performed to meet those goals was running with a team – joining a development team for 1-2 months to produce high quality results together, and to build quality into the product. 

As an experienced software tester, this was the quality coaching activity which felt most familiar and comfortable for me. I worked as part of an agile team to develop acceptance criteria, and helped guide the team on creating a test strategy and approach to suit their project. We used Atlassian’s quality assistance model as a starting point and adapted the process over time to meet our needs.

QA Kickoff

For individual stories I would attend a QA Kickoff meeting to map out test scenarios with the developer and product manager. During the meeting I’d ask a lot of questions, and together we would consider risks, probe edge cases to help determine scope, and consider various relevant aspects of software quality. Most of the mind map contents were generated together during meetings so that sessions were more collaborative and creative, to encourage thought and input from different perspectives. 

Initially when running with a new team I’d try to ask specific, leading questions during the QA Kickoff, similar to the questions asked by testers. Then over time I would slowly switch to coaching mode, asking broader questions to promote a longer-lasting quality mindset.

Software Testing QuestionsQuality Coaching Questions
What will happen if 2+ users click this button at the same time?Do you have any concerns around concurrency?
Together, can we review the tests which cover these risks?
Will data integrity be maintained if this process is interrupted? Are there any risks to data integrity?
How are they being mitigated/prevented?
Which file types are unsupported?
Is an error displayed when an unsupported file is selected?
Should the error message contain more details?
What can go wrong?
How will the user know something is wrong?
Will they have enough information to fix the issue?
I’ve added a UI automation test for this bug. How can we prevent this type of bug from reoccurring?

The software testing questions remind me of coaching sessions with junior testers in past roles. By asking specific questions, you’re leading them towards a quality risk area which you’ve already identified. The quality coaching questions remind me of reviewing a test plan prepared by a senior tester. You ask more open questions and then work out the answers while learning together.  

Unlike a software tester, the quality coach then relies on the developer to perform the actual testing and provide answers to the questions raised during the QA Kickoff meeting.

QA Review

For faster feedback loops, the developer should test their own changes prior to the QA Review. These meetings can also include a product manager, a senior/peer developer and a designer. The developer demonstrates that all scenarios from the QA Kickoff have been covered, and explains any changes made to scope/functionality since the QA Kickoff meeting. The quality coach can provide coaching during the session, usually in relation to test coverage gaps and automation test code patterns. 

I’ve found these sessions to be effective in helping developers shift perspective from the lines of code back to the user experience, and to look at the bigger picture.

But “developers can’t test their own work, can they?

To be blunt, developers being unable to test their own work is a fallacy that has suited the software testing industry very well. It has also been a convenient theory for those software developers who prefer to let someone else do their testing for them. My own opinion on this has changed in recent years, and it’s sheer luck if I don’t have a poorly-aged blog post stating that developers can’t test their own code.

I’ve seen that the best developers can and do test their own work. In an unhealthy culture, those developers can be viewed by their peers as slow and less productive, despite having better code quality and less bugs found by testers, and therefore less rework required. In those unhealthy environments, the testers and engineering managers have different ideas about which developers are the more valuable and productive members of the team. Senior testers, when paired with developers who test their own code, are able to adopt more of a quality coaching mindset and help the developer improve their own testing skills and code quality.

Quality coaching mindset

Developers on your team may be used to working with testers and be uncertain of what quality coaching means for them. It helps to clearly describe and demonstrate what quality coaching involves, and specifically what it doesn’t include – i.e. testing their changes for them.

Having a quality coach work with the same team for too long can blur the lines between quality coach and software tester. I found that by changing teams regularly, it reinforced the message that the whole team is responsible for quality. It also reminded me to act as a quality coach, especially at times when I was tempted to jump in and do some testing myself. It encourages a subtle focus shift from having product quality as the only goal, to helping the team reduce their feedback loops and independently produce quality software.

Knowing that there are no testers in the team, and that the quality coach will be back on the team again in a few months, hopefully helps to encourage a lasting quality mindset amongst the developers in the team. 

This is Part 3 in a short series of posts about quality coaching. The next posts describe some additional activities performed by a Quality Coach. 

 
Leave a comment

Posted by on August 15, 2021 in Software Testing

 

Tags: ,

The Role of a Quality Coach

Context 

For eight months I’ve worked with software development teams in the role of senior quality coach. We work in agile teams, without ever using the word agile. Most teams include software developers, a product designer, a product manager and an engineering manager. There are no testers or business analysts, and everyone on the team can write code to some degree. 

Quality coaching is not software testing

Quality coaches do not fill the role of software testers. While it’s commonly said that each person in an agile team owns quality, this truth can be undermined by the presence of a tester in the team.*

To state it more clearly: quality coaches do not own the test strategy, automation test suites, build pipelines, lists of defects, test environments, or test data. Quality coaches are not responsible for investigating failed test results, performing exploratory testing, or writing up bug reports. 

Every one of these artefacts and activities is owned and maintained by the development teams. 

Quality Coach Responsibilities

Create and promote a quality mindset within the team. 

Work with stakeholders to discover/determine quality goals. 

Explore the product to gain context. 

Provide coaching, guidance and training on software quality and testing.  

Collaborate with other teams including Support, Sales and Marketing to better understand user goals and behaviours.

Stay curious, ask questions, follow up on loose threads. 

Help facilitate risk workshops, bug bashes, post-mortems and other brainstorming sessions.

Advocate for process improvements.

This is Part 2 in a short series of posts about quality coaching. The next posts elaborate on the activities performed by a Quality Coach. 

*Embedding a tester in an agile team can work very well, for example when the bulk of testing is performed while pairing with the developer who wrote the code.

 
Leave a comment

Posted by on July 11, 2021 in Software Testing

 

Tags:

Coaching Testers vs. Developers

After many years leading and coaching testers, for the past 6 months I’ve been working with developers as a quality coach. I’m keen to share lessons learned from my experience so far and to learn from others working in Quality Assistance and Quality Coaching roles.

What’s different?

Enthusiasm

Testers are eager to learn more about testing, it’s their passion. The “pull method” of providing coaching and training on demand worked very well, where I made it known that I’m available to the team and they approached me to arrange sessions. Testers reached out to ask for advice, and I was invited to meetings, asked to consult or collaborate, or asked to review work. In some cases my reputation in the testing community has preceded me to a new role, and testers were keen to collaborate from the moment I joined the company.

Great developers are also open to learning more about testing, but the word ‘enthusiastic’ doesn’t spring to mind. If I was to rely solely on the pull method of waiting to be asked for coaching, I would have a lot of spare time on my hands. Instead I attend standup meetings, planning sessions and retros, and I look for opportunities to propose coaching and pairing sessions. It requires me to be more confident and assertive, and somewhat persistent. It’s fair to say that my reputation doesn’t precede me with developers!

Arranging the first session with each developer is the biggest hurdle. The quality assistance/coaching model is relatively new for most people, and there’s uncertainty around what will be involved. As with most things, it gets easier each time.

Training method

For a team of testers, I could present three new concepts in three weeks. For example: mind-mapping test scenarios, focus/defocus exploratory testing technique, and consistency heuristics. Testers love learning these techniques, and incorporating them into their regular work. I picture them adding new tools to their toolbox, to help them find quality issues more efficiently and effectively. The value and usage of each new tool is self-apparent, meaning it’s obvious to the testers where and when each new concept will be useful.

Working with developers, any coaching on testing methods will be more successful if demonstrated and proven in the context of the code/feature/product they’re currently working on. I’ve found that I’m not able to update and reuse my existing presentations and materials which are theory-based and use practise testing websites. It’s almost as though the method you’re demonstrating needs to find a bug in your own product during the training session, to be considered a technique worth learning. I picture developers with a small extra toolbox for testing that’s already full, and they’re not convinced yet they need to purchase a larger toolbox!

My own expertise

When doing one-on-one coaching I learn new things every single time, such as domain knowledge, keyboard shortcuts, useful browser extensions, etc. Even so, I’m typically guiding more than I am learning.

While coaching developers it feels more like a two-way learning process. While asking leading questions about quality, scope, and risk, I’m also learning about the operating environment, debugging methods, code structure, in-house test tools, and more. Importantly, I’m seeing how many bugs can be traced back to code patterns and structure, and can therefore be prevented at that level during development. It’s exciting to learn new ways of preventing issues rather than detecting them later.

Which is better?

When I’m working with testers I automatically assume the role of mentor, trainer, leader. For years this has been my comfort zone – testers are my tribe, my people, my community. Personally and professionally, I thrive when I’m just outside my comfort zone. In the past that has lead me to pursue consulting, contracting, senior management roles, public speaking, organising meetups, hosting training courses… Now taking on this new role has allowed me to use my experience in the software industry with quality and coaching, while pushing me to learn about development more deeply.

I’m hopeful that by coaching developers I can have a greater impact on preventing/reducing product quality issues and therefore help to produce quality software faster. It’s still early days.

Please reach out if you’re also in a Quality Coach role and would like to share notes.

Part 1 in a short series of posts about Quality Coaching.

 
Leave a comment

Posted by on June 13, 2021 in Coaching, Growth, Learning, Quality

 

Tags: , , , ,

How did you miss that bug?

How did you miss that bug?

That tense feeling we get when users find an important issue in production.
The panic when a manager asks the dreaded question.
Searching through your own notes and test results wondering, how did I miss this…?
Eventually, taking some deep breaths, you remind yourself that exhaustive testing is impossible.

However, you’re not entirely off the hook. As a quality champion for your team you can help to identify risk areas in the software development process and suggest improvements. To bolster your case for change, explain how each improvement could have helped to prevent recent production issues, or at least detect them sooner.

Some recent examples from my own experience include:

  • Test environment was configured differently to the production environment, hiding the issue (i.e. no load balancer).
  • The telltale sign in the log files was missed during testing due to the amount of noise in the error logs and lack of monitoring tools.
  • Test data was too far removed from production data, so the problem wasn’t apparent.
  • Automated test coverage was lacking in some areas, due to limitations in the framework.
  • Some important legacy code was lacking unit tests, which I hadn’t realised.

It’s important that you don’t come across as making excuses, or passing on blame to others. Instead, each point can be used to drive improvements. For ideas on further improvements without waiting for production issues to occur, read The Science of DevOps: Accelerate by Forsgren, Humble and Kim. This book contains excellent ideas for improvements to your team’s processes, backed by data demonstrating links to increases in business performance.

Exploratory testing can help to find issues before they go to production, particularly with edge case scenarios. Exploratory testing is a skill that needs to be learned and practised. There are cheat sheets, heuristics and TestSphere cards to help you find important bugs fast.

Finally, and perhaps most controversially, how much does it matter if some bugs make it through to production? Consider the potential risks of failure; the effectiveness of monitoring and alerting tools used in production; the time required to fix production issues; whether a phased rollout is possible… All of these factors should affect the overall development, testing and deployment approach.

Asking how the testers missed a bug is simply not good enough. Expect more from your managers and your agile team members. In a healthy work environment, the team may ask instead, “How can we all help to prevent this type of issue from occurring in the future?”

 
1 Comment

Posted by on February 29, 2020 in DevOps, Software Testing

 

Are You Adding Value, Always?

How do you add value for your company, your team, your customers, and product quality?

The exercises below will help you to focus on adding value through your work, and to confidently describe your contribution to stakeholders.

Analyse and Evaluate

For two weeks, track the tasks you’re working on each day. How is the bulk of your time being spent? The answers may surprise you and your team.

When performing this exercise for myself or with a team member, I keep track of tasks in a mind map as shown below (created using Mindmup). You could also use a spreadsheet.

Value1a.JPG

Figure 1. Example mind map for tracking tester activities

It’s incredibly helpful to review this with your manager. Together you can develop a plan to complete high-value tasks more efficiently, and make low-value tasks obsolete.

To help evaluate the value of each task, consider the following:

  1. Why did you do it?
  2. How did it contribute to your goal of delivering high-quality software?
  3. Is this task genuinely necessary? What is the cost of doing this versus the cost of not doing it?
  4. Could the same outcome be achieved more efficiently?
  5. Could a better outcome be achieved in another way?

Be Efficient and Effective

Next, map each task that you’ve identified to the Cost-Benefit diagram below. 

Value2

Figure 2. Consider the relative cost and benefit of tasks

For example, if you’re producing lengthy documents and can confirm that nobody is reading them – stop creating them. Or, if stakeholders are reading only certain sections of your documentation, then those are the only sections you should be producing.

If another task is relatively high cost and high benefit, look for ways to improve efficiency. For example, script repetitive tasks such as configuring test environments, creating test data, or monitoring log files for errors.

To further improve my own effectiveness at work, I try to minimise distractions for a period of time each day. I close email and chat programmes for at least one-hour twice per day, and focus on my other work. Otherwise, I find it difficult to add value in the midst of constant distractions and context-switching. It’s good to experiment and find a time-management method which works for you.

Opportunity Cost

Consider the opportunity cost of activities being performed – by doing any task, you are giving up the opportunity to perform other tasks during that time. Let’s use an example of a manual regression testing cycle which takes three hours to complete. This means having three hours less to spend on activities such as testing user stories, reviewing customer analytics data, or pairing with a developer.  

As professionals, we have a duty to understand the value and opportunity cost of our activities. As an example, manual regression testing cycles are often a lot longer than necessary, and it pays to review them regularly. When I join an organisation and review existing manual regression testing processes, it’s clear that high-impact issues are found during the first 10% of the testing time.

“If you always do what you’ve always done, you always get what you’ve always gotten” – Jessie Potter¹

Perspective

It’s important to be on the same page as stakeholders regarding the relative cost and benefit of regular tasks.

If you’re required to complete a low-benefit task, perhaps you’ve misunderstood why that particular test activity or document is needed. Ask more questions about the purpose, benefit, and value of the requested deliverables.

If a task seems very expensive, perhaps the cost of doing that task hasn’t been clearly communicated and understood by your stakeholders. Provide high-level information about the steps involved and the time required.

Do you provide metrics which require considerable manual effort to collate? Stakeholders may assume that these figures are readily available.

Do you copy and reuse test documentation and make only minimal changes to the new version? If so, there’s a good chance that nobody is reading these low-value test artifacts, and they’re probably not needed at all.

Work together with your stakeholders to achieve the best outcome. Try to reach a compromise so that you’re only producing deliverables which are valuable, while still providing all of the information required. Then produce those deliverables as efficiently as possible.

Influence

As a test professional, it’s part of your responsibility to speak up when asked to do something unproductive. However, pick your battles wisely and start with the highest cost and lowest benefit tasks first. Always offer potential solutions to stakeholders and team members where possible, not just complaints.

It’s important to have a clear understanding of what your manager and your team expects of you. Demonstrate how much of your time is being spent on those goals versus distractions or busy-work with little benefit. Stakeholders understand Return-on-investment (ROI) and cost-benefit analysis terminology. Outline the ROI of planned improvements to testing processes, to help gain support for innovation and change.

The majority of test professionals in Australia and New Zealand are fortunate to have some influence over their own test approach. However, testers working in a less-agile/ more-bureaucratic organisation may have less opportunity to improve their way of working.   

Learning new skills is one way of adding value when you have limited influence. You can learn in-depth about the tools you’re using daily, and discover features which can save you time. Another suggestion is to read online about new test techniques. Remain focused on delivering value to your company and your customers.

Add Value – Always!

The relative cost and benefit of a particular task will change over time. It’s useful to revisit this exercise regularly, e.g. quarterly. The expected outcome is that you’ll be working on the right tasks at the right time, in an efficient and effective way. You’ll be confident in describing your role and the value you add to the team.

After all, if you can’t articulate the value you’re adding, then how will others know?  

——————-

¹ https://quoteinvestigator.com/2016/04/25/get/

 
2 Comments

Posted by on September 11, 2018 in Software Testing

 

The Siren Call of Test Metrics

The Siren Call of Test Metrics

This article was originally published in Testing Circus.

Reporting on software testing using metrics can have unintended consequences. Which test metrics do we regularly produce for reporting, and what information will stakeholders interpret from those metrics? Let’s examine some real-world examples in detail.

 

95% Tests Pass, 1% Fail, 4% Untested

Test pass/fail metrics are popular in traditional test progress reports. Consider for a moment that you are the product owner and you have final responsibility for deciding whether to release a product. The test manager presents you with this graph:

Test Progress chart showing 95% pass, 1% fail and 4% untested
There’s certainly a lot of green in this graph, and green is usually good. So, does this graph make you feel more inclined to release the product?

When I read this graph I have more questions than answers. For example:
Failed tests
– Which tests have failed?
– What was the cause of failure and what’s the severity and status of the related bugs?

Untested tests
– Which tests have not yet been tested?
– What’s the risk of not doing those tests?

Passed tests
How well have the major risk areas been tested?
How were those major risk areas identified?

 Risks
– What else has not been tested, aside from whatever is represented by the graph, that the test team feels should have been tested? Examples include user scenarios, end-to-end tests, error-handling, performance, security testing and so on.
– What is the test team’s confidence level in product quality?

While the above graph looks more promising than another showing 50% pass and 50% untested, it’s still only a small part of the overall testing story. When viewed in isolation this graph is virtually meaningless, and can provide stakeholders with a false sense of confidence in product quality.

100% Requirements Coverage

This metric sounds reassuring, and looks impressive when presented as a pie chart.

Requirements coverage pie chart showing 100% green

What information is represented here? The data source for this metric is usually a requirements traceability matrix (RTM). A set of tests are compared to a business requirements document (BRD), and every requirement is ‘covered’ by at least one test. The RTM does not – and cannot – take implicit requirements into account, it is limited to explicitly stated requirements.

Some teams report 100% requirements coverage without ever having executed a test, as the RTM matches tests to requirements irrespective of whether those tests have been performed. A team on a traditional project who have painstakingly written tests for each requirement – without ever having seen working software – may report 100% requirements coverage based on the BRD.

During the development and testing cycle, features invariably start to differ from the original requirements. This may be due to implementation limitations, de-scoping of features, changes to the design, and so on. Requirements and written tests quickly become out-of-date. At that point, how useful is this metric?

On a hypothetical highly-organised and well-documented project, if the requirements and tests are constantly kept up-to-date and we perform all of the planned tests, then can we report on 100% requirements coverage? The RTM shows which requirements the testers consider to be covered by at least one test case. It doesn’t show the extent to which the requirement will be tested. As we see in this tutorial from Cem Kaner, 100% test coverage of the system is impossible to achieve in any realistic software testing scenario. We cannot test every input, every combination of inputs, every path through the code.

If you report on this metric do so with caution, and with multiple disclaimers included in your report. Again, this metric can provide a false sense of confidence to stakeholders.

12 Bugs Raised Today

The number of bugs raised per day metric can be enhanced by providing more details, such as a breakdown by severity and trends over time. The major risk in simply reporting this data to stakeholders in a graph is that it may be interpreted in any number of ways.

Graph showing bugs raised each day grouped by severity

Here are some real-world interpretations of this metric from different stakeholders:

  • Developers aren’t testing their own work enough, “Testers are finding too many bugs”
  • Testers should have found these bugs sooner, “Why was a Sev1 only found yesterday?”
  • Testers are working hard at the moment, “They’re raising lots of issues”
  • Testers aren’t working hard enough, “They could be raising at least 20 bugs every day”
  • Product quality is poor, “Look how many bugs were raised in the last two weeks”
  • Product quality is improving, “There’s less bugs now”

Some of these interpretations may coincidentally be true, but could just as easily be incorrect. If your project stakeholders are presented with a graph similar to this one, are they likely to ask for more information, or leap to their own conclusions?

If testers notice stakeholders equating high bug volume to increased testing effort, how might that affect their behaviour? For example, if a tester in this situation found three spelling errors on one page, they might decide to raise them all individually to increase the volume of bugs raised that day. Then there would be an increase in overhead to process those through the bug tracking system. This in turn could be seen as wasting other people’s time, and fracture relationships between the developers and testers.

0 x Sev1, 3 x Sev2 Bugs Outstanding

There are no showstopper Severity 1 bugs waiting to be fixed for this release, and only three critical Severity 2 bugs.

This metric is a great conversation starter, leading to the obvious question: What are the three Sev2 bugs and do they need to be addressed before the product can be released?

There are some less obvious questions to consider:

  • Has the severity of the bugs been reviewed by more than one person?
  • Are these bugs regressions from the current released version?
  • Have the high-risk areas of the product been tested?
  • What other bugs are outstanding?
  • Were any Sev2 bugs downgraded to Sev3 for the purpose of producing this metric?

Focusing only on the high severity bugs may mask high volumes of minor bugs. If there are a group of minor bugs relating to a particular feature or aspect of the product, they can amount to the equivalent of a critical issue. For example, a number of minor bugs in the back-end UI caused staff to spend longer on tasks which they performed frequently, slowing down overall productivity and causing frustration with the system.

Misuse of Metrics

For the most part, reporting on these metrics alone is the sign of an inexperienced, frustrated or unempowered test lead, and is done with good intentions. I know test leads who produce the metrics requested of them (usually by project managers) while feeling frustrated that the metrics don’t accurately reflect their opinion of product quality, because metrics only tell part of the whole story.

A real-world example of this is a project which had at least one test per written requirement, 95% of tests were passing, and there were no Sev1 bugs outstanding. Based on those metrics, the project manager was satisfied that the exit criteria had been met, and that no further time was needed for testing. As we’ve seen above, each of those metrics is flawed if taken in isolation. In this case, the test manager had an uphill battle explaining why further testing time was required to test the major risk areas more thoroughly, while all metrics graphs were essentially reporting a green light.

I have seen a case of metrics being produced which were deliberately misleading. On one project I joined, the previous test manager had asked the team to prepare two test cases per business requirement: one positive test case and one negative test case. Here’s an example of a written requirement from that project:

Staff access to the system is limited according to their role.

It’s entirely possible to write two statements which appear to cover this requirement:

  1. Verify that staff users can access the system functions they should be able to perform 
  2. Verify that staff users can’t access system functions which they shouldn’t be able to perform

While the above requirement is clearly lacking in detail, such as which functions the users should and shouldn’t be able to perform, it’s a real-world example. (The BRD had even been reviewed and signed-off by project stakeholders.) Rather than raising a red flag, the test manager had prepared test artefacts based on the limited information available at the time, and had traced those back to the requirements. This masked the fact that more information was needed before this requirement could be tested.

The test manager was reporting 100% requirements coverage before test execution had begun, based on the RTM. Having now read just one requirement and the associated coverage of that requirement, how much value would you place on that metric? What do you anticipate will happen when the test team attempt to verify whether this requirement has been met by the system?

Moral of the Story

Test metrics and graphs can enhance the testing story, but cannot replace it. Do not present or judge testing progress by test metrics alone, use qualitative information as well. Speak with the testing team regularly to learn the whole story.

These graphs are like the siren songs of Greek mythology. They are appealing, hollow, and will lead to a great deal of sadness and destruction if followed unquestioningly. For alternative reporting methods, see the Reporting on Test Progress section of my article in the Testing Trapeze 2015 August Edition.

 
9 Comments

Posted by on February 4, 2018 in Software Testing

 

Tags: ,

What Does Testing Look Like Without Test Cases?

This article was originally published in Testing Trapeze.

At testing meetups, workshops and conferences, context-driven testers are encouraging others to move away from relying solely on test cases. We reason that the prescribed use of test cases and test metrics is an inadequate approach to software testing. We believe that responsible testers must constantly adapt their approach to meet the current needs of the project.

After these events I hear feedback from attendees who are inspired, yet don’t feel that they could move away from using test cases on their project. Some of them anticipate objections from management, and struggle to come up with responses to explain this different testing approach with confidence. The objective of this article is to offer insight into performing software testing and progress reporting effectively, without relying on test cases, and to help you sell and defend this approach to your stakeholders.

Why not use test cases?
As a contract test specialist I’ve had the opportunity to work on a wide variety of projects. Sometimes I join at the start of a project, but more often I’m hired as the hands-on test manager of an in-flight project, with existing test processes, test cases, and a tight deadline.

All of the test cases I’ve ever inherited have been out-of-date – without exception. They were initially written based on requirements documents. As the project details changed, the time\effort needed to keep test cases updated was better spent on testing new features and bug fixes as they were developed. You might have noticed that your own test cases are out-of-date and time-consuming to maintain. Perhaps you’ve wondered if the time you spend re-writing test case steps would be better spent on other testing activities.

At their best, test cases state the obvious to testers who are already familiar with the product. At worst, test cases contain incorrect or obsolete details which lead new testers astray, and require updates which slow down experienced testers like the proverbial ball-and-chain. Let’s look at three common test activities, without the use of test cases: Test preparation, execution and reporting.

Test preparation
As with traditional projects, I map out a test strategy and/or test plan. What’s different is my approach to them. Instead of filling in templates with standard details common to all projects, I create a custom document or mind map to suit the project, using the Heuristic Test Strategy Model (HTSM) as a guide. I constantly reflect on my actions, and ask myself whether the document\section\paragraph is adding value to the project, and whether it is useful and necessary.

So where’s the harm in writing test cases during the test planning phase? The harm isn’t necessarily at the moment of creation; the cost of creating a test artefact also includes the cost of maintenance. Without maintenance, test cases quickly become outdated. Consider whether the information is already available somewhere else, for example, in a requirements document or user stories. It’s more efficient to keep information updated in a single location, rather than duplicating the information across artefacts and across teams.

Instead of spending substantial amounts of time creating difficult-to-maintain test cases, spend this time on other preparatory activities. Read all available documentation, get familiar with existing company products, explore earlier versions of the product and competitor offerings. Improve your product domain knowledge, learn about business processes – present and planned. Talk to developers about the technology stack. Talk to BAs about their design decisions. As you learn, jot down test ideas. This could be in the form of a mind map, checklists, test charters… anything lightweight which can be easily revised.

Test Execution
Working on more-or-less agile projects, I’m focused on testing new features as they’re developed, and keeping up with the pace of development. I’ve found that there’s never a shortage of things to be tested.

My main priorities while testing are to verify that explicit and implicit requirements are met, find bugs and report them. To do this effectively I use a form of structured exploratory testing called thread-based test management (TBTM). This is similar to session-based test management, but testing activities follow a set of ideas rather than being time-boxed. The majority of test ideas are created while testing. I have a Microsoft OneNote file per testing thread, with a new tab each time I retest a feature. Most of my testing threads are feature-based, corresponding to user stories.

When retesting bug fixes, I use a micro-TBTM approach. I treat each bug as a separate thread and enter my test ideas and results directly into the bug tracking tool as a comment. Anyone tracking that bug will be notified of my change, and can provide feedback or ask questions.

For example:

Passed testing in build 33, test environment 1.

  • Postcode field is now populated automatically when suburb is selected – Pass
  • Postcode cannot be manually entered or edited – Pass
  • Still unable to proceed without entering a valid suburb – Pass
  • Typo in validation message, raised separately Bug 2323
  • Suburbs which are duplicated in other states “Silverdale” – Pass
  • Suburbs containing a space in their name “Cockle Bay” – Pass
  • Suburbs which have more than one postcode “Hamilton, NZ” – Pass
  • AU/NZ suburb names containing non-alpha characters – Couldn’t easily locate any, not tested
  • Tested in latest versions of Chrome, Firefox, IE and Safari
  • Tested on desktop and mobile site
  • Updated automated check for customer sign-up

TBTM is traceable, auditable and efficient. It’s great to also have [heuristics] checklists to compare test coverage with, particularly if you start running out of ideas for things to test. I find it helpful to run through my checklists prior to release, to gain confidence in my test coverage.

Reporting on test progress
A lot of my test reporting is verbal, for example, during daily standup meetings and in response to direct questions. I provide weekly test progress reports to stakeholders to reiterate these verbal reports, starting from the initial test preparation stages. The information in these weekly reports feeds into the test summary report at the end of the project.

I’ve learned a powerful secret in my quest to stop reporting on test-case metrics: Senior management appreciate bullet points almost as much as graphs! So I keep my test reports short, useful and relevant.

Below is an example weekly test status report, which lists up to five points each for the most important outstanding product issues, risks, tasks completed this week, obstacles\blockers to testing, and objectives for next week. Consider which headings you would add or remove for your project:

Top issues:
Registration failing in Firefox – Bug 1234
User remains logged in after performing logout – Bug 1235
– Unable to renew accounts before the account expiry date – Bug 1236

Risks:
Performance test tool license expires next week, payment is still pending approval

Achieved this week:
– Focused on testing new features being developed: registration, login\logout and account page
– Briefly tested renewals and user profiles: no critical issues raised
– Performed thorough cross-browser and cross-device testing of front-end UI

Obstacles\blockers:
– 2 testers were sick this week
– Test environment data refresh delayed until next week

Objectives for next week:
– Continue testing newly developed features and fixes
– Review performance test plan

Any of these points can be a conversation starter, providing you with the opportunity to tell the testing story in more detail. Given the opportunity, you’ll be able to discuss coverage and depth of testing, and perceived product quality. Don’t feel constrained by the questions you’re being asked. For example:

Project manager: What percentage of test cases has passed?
Test lead: That’s not useful information. Here’s what you need to know…

Adding some metrics to your reports is a valid option. For example the ‘number of resolved bugs waiting to be retested’ or the ‘number of bugs raised this week by severity’. However, be aware when including metrics that you run the risk of readers misinterpreting the figures, or placing more importance on the metrics than the rest of the report.

I like to present test coverage and quality perceptions to stakeholders in a visual model, for example as a heat map. Heat maps can be created with mind mapping software but recently I’ve switched to using tables, primarily because stakeholders are more familiar with using spreadsheets and don’t require additional software to be installed on their computers. Here’s an example:

heatmap table

In this example, the green cells represent a working feature, red indicates feature not working, orange is a feature that works but has issues, grey is untested and white is unsupported. So, at a glance this heat map shows:

  • which areas of the product have and haven’t been tested;
  • a list of supported features per platform;
  • test team impression of quality per feature; and
  • the corresponding issues or bugs.

Introducing context-driven testing
At this point you may have noticed something interesting: all of the above methods can be used alongside test cases. Sometimes this is how I introduce context-driven testing to a team; gradually. Each time, my team has found that as the project progresses, there is less and less value in referring back to their old test cases.

A few weeks after making the switch to context-driven testing on one such project, some managers became nervous that we might be missing something by not executing the test cases. The other testers and I started to wonder if they might be right, so we tried an experiment. Half of the team spent the next three days executing test cases, and the other half of the team continued with structured exploratory testing. We met at the end of the three days to review our findings.

Even I was surprised to find that the testers executing test cases had not found a single new bug. They felt frustrated and constrained by the outdated test cases, and had put pressure on themselves to get through the test cases as fast as possible. A large chunk of their time was spent looking up existing bugs they had raised, in order to add the issue number reference to their failed test case results. We also realised that they had hardly spoken to a developer or business representative.

Less surprisingly, the half of the team performing structured exploratory testing found the exact opposite. They had continued to work closely with the project team, raising bugs and progressing with feature testing coverage. After that we agreed to leave the test cases in favour of using the business requirements document as a checklist, to confirm whether we’d missed testing any important product features.

Conclusion
There is no one-size-fits-all “best-practice” approach to software testing. I have implemented a new context-driven testing approach for each project I’ve worked on in the last 18 months, across different teams, companies and industries. I’ve earned the respect of project managers, testers, developers and business stakeholders. The turning point for me came after Let’s Test Oz 2014, when I took Fiona Charles’ words to heart – “I’m not going to do bad work”.

The first time I refused to use existing test cases I was nervous. I knew the principles behind context-driven testing and I was keen to implement change. I also felt that I was going out on a limb and putting my professional reputation on the line. Yet it was surprisingly easy to demonstrate the value of context-driven testing approaches to stakeholders, and to gain their support.

Convincing teams to adopt context-driven testing approaches became easier over time. As I built up a portfolio of success stories my confidence improved. I don’t expect junior testers to have this capacity for change within existing project teams. The responsibility lies with test managers and test leads to demonstrate to project stakeholders what software testing really involves.

Context-driven testing is not easy and it’s not boring. It’s challenging, rewarding and – best of all – it’s a more effective way to test. Testers following these principles are more motivated, engaged, valued and respected.

Remember that testing is about providing valuable information, not just the percentage of passed test cases.

With thanks for inspiration:

 
5 Comments

Posted by on June 29, 2017 in Learning, Software Testing

 

One Tester’s Account of an iOS Update

One Tester’s Account of an iOS Update

As a tester I’m well aware that things can go wrong during software updates. I always wait at least a week before upgrading iOS on my iPhone, leaving time for issues to be reported and fixed. When I run the upgrade, I’m expecting something to go wrong. This means that I always backup my phone first – luckily!

 

During the latest upgrade, I saw this on my iPhone:

 

iphone6-ios9-recovery-mode-screen

I’d been expecting something to wrong and my iPhone cable is looking a bit worse-for-wear these days, so I assumed that my phone had been disconnected. Doesn’t that screen look like it’s asking me to plug in my phone?

 

Naturally I unplugged and plugged in the cable again to restore the connection.

 

In hindsight I think that image means “iTunes is updating your phone, do not under any circumstances unplug the cable”. From a usability perspective, this image is ambiguous at best. Poor usability combined with my pessimism caused the update to fail.

 

I’ve had only one iOS update fail in the past, around 2 years ago, yet since then I pretty much expect them all to fail. A catastrophic failure can leave a lasting impression of poor software quality.

 

A popup dialog said that I needed to run the Restore next which would clear all of my phone settings and restore from backup. That sounded straightforward to me, because I’ve restored my phone in the past with no issues (due to hardware upgrades and replacements). Based only on my personal experience, my impression was that the Restore feature is more reliable than the Upgrade feature. In fact, the upgrade has failed for me just once out of many uses, while the restore has always been successful but I’ve only used it four times.

Then the restore from backup failed, with Unknown error 21.

 

error_tweet

The dialog with the unhelpful wording led to a website with clear instructions. Some steps were less helpful than others, e.g. Upgrade Windows and reboot your PC.

 

After following the steps I tried again, and this time the Restore process looked like it was working.

 

Then iTunes asked for my backup password. I’d just created a backup this morning and hadn’t been asked to give it a password…? I tried my Apple ID password, with no luck.

 

I found an Apple forum post where someone was complaining about having no way to recover the password. The Apple fan club had gone the offensive, “How could you forget your password? This is your own fault!”. The defence was, “How can I be expected to remember my password from years ago when I bought my first iPhone?”

 

Aha! A clue… I tried one of my old passwords from days gone by and it worked! The person complaining was more help to me than the Apple-defenders.

 

My iPhone is now restoring from backup and I’m breathing a sigh of relief. I feel like it could be smooth sailing from here, and there’s only 22 mins remaining.

Now 27 minutes.

Now 34 minutes.

 

Hmm…

 
Leave a comment

Posted by on July 30, 2016 in Software Testing

 

WeTestAKL Meetup 99-sec Talks

Our July WeTest Auckland Meetup came with this warning:

If you attend this meetup, you will be speaking!

Inspired by TestBash, Shirley Tricker organised a meetup where everyone was required to speak.. but only for 99 seconds!

We started with two 25-minute talks from first-time speakers, on mobile test automation and communication skills for testers. Then we launched into the 99-sec talks.

Shirley chose four people at a time to come to the front and speak for 99 seconds each, to a group of 16 testers. Some people had clearly practiced their timings, delivering their closing sentences just seconds before the buzzer. Many more were surprised when the buzzer sounded mid-sentence!

I’m so impressed that everyone who came along spoke: including people who’d made it off the waiting list just one hour before the meetup; some first-time attendees; plenty of first-time speakers; and those who hadn’t read the Meetup properly and didn’t realise that they needed to talk!

 

Lauren Young- What is a Statement of Work?

Kasturi Rajaram- Why I love Non-Functional Testing

Rasha Taher – HiPPO: Highly Paid Person’s Opinion

Sandy Lochhead- Child’s Play, the Value of Playing Games

Nadiia Syvakivska – Luck

James Espie – Maintaining Perspective

Reshma Mashere – Listening Skills and Meditation

Monika Vishwakarma – Thank You WeTest

Bede Ngaruko – Fear of Failure

Jennifer Haywood – Mobile Testing Mnemonics

Inderdip Vraich – Specflow

Nav Brar – My Journey to NZ

Shirley Tricker – Career Tips

Laurence Burrows – Raising Defects

Jennifer Hurrell – Documenting Exploratory Testing

Kim Engel – #30DaysOfTesting Challenge

Thank you Shirley for creating such a friendly and fun environment where we could share and learn. And thank you to our sponsors for the evening, Assurity.

* More details on the 99-sec format:
http://scottberkun.com/2012/99-second-presentations/
http://www.ministryoftesting.com/2015/06/99-second-talks-go-virtual/
 
1 Comment

Posted by on July 30, 2016 in Software Testing

 
 
%d bloggers like this: