The Role of a Quality Coach


For eight months I’ve worked with software development teams in the role of senior quality coach. We work in agile teams, without ever using the word agile. Most teams include software developers, a product designer, a product manager and an engineering manager. There are no testers or business analysts, and everyone on the team can write code to some degree. 

Quality coaching is not software testing

Quality coaches do not fill the role of software testers. While it’s commonly said that each person in an agile team owns quality, this truth can be undermined by the presence of a tester in the team.*

To state it more clearly: quality coaches do not own the test strategy, automation test suites, build pipelines, lists of defects, test environments, or test data. Quality coaches are not responsible for investigating failed test results, performing exploratory testing, or writing up bug reports. 

Every one of these artefacts and activities is owned and maintained by the development teams. 

Quality Coach Responsibilities

Create and promote a quality mindset within the team. 

Work with stakeholders to discover/determine quality goals. 

Explore the product to gain context. 

Provide coaching, guidance and training on software quality and testing.  

Collaborate with other teams including Support, Sales and Marketing to better understand user goals and behaviours.

Stay curious, ask questions, follow up on loose threads. 

Help facilitate risk workshops, bug bashes, post-mortems and other brainstorming sessions.

Advocate for process improvements.

This is Part 2 in a short series of posts about quality coaching. The next posts elaborate on the activities performed by a Quality Coach. 

*Embedding a tester in an agile team can work very well, for example when the bulk of testing is performed while pairing with the developer who wrote the code.

Leave a comment

Posted by on July 11, 2021 in Software Testing



Coaching Testers vs. Developers

After many years leading and coaching testers, for the past 6 months I’ve been working with developers as a quality coach. I’m keen to share lessons learned from my experience so far and to learn from others working in Quality Assistance and Quality Coaching roles.

What’s different?


Testers are eager to learn more about testing, it’s their passion. The “pull method” of providing coaching and training on demand worked very well, where I made it known that I’m available to the team and they approached me to arrange sessions. Testers reached out to ask for advice, and I was invited to meetings, asked to consult or collaborate, or asked to review work. In some cases my reputation in the testing community has preceded me to a new role, and testers were keen to collaborate from the moment I joined the company.

Great developers are also open to learning more about testing, but the word ‘enthusiastic’ doesn’t spring to mind. If I was to rely solely on the pull method of waiting to be asked for coaching, I would have a lot of spare time on my hands. Instead I attend standup meetings, planning sessions and retros, and I look for opportunities to propose coaching and pairing sessions. It requires me to be more confident and assertive, and somewhat persistent. It’s fair to say that my reputation doesn’t precede me with developers!

Arranging the first session with each developer is the biggest hurdle. The quality assistance/coaching model is relatively new for most people, and there’s uncertainty around what will be involved. As with most things, it gets easier each time.

Training method

For a team of testers, I could present three new concepts in three weeks. For example: mind-mapping test scenarios, focus/defocus exploratory testing technique, and consistency heuristics. Testers love learning these techniques, and incorporating them into their regular work. I picture them adding new tools to their toolbox, to help them find quality issues more efficiently and effectively. The value and usage of each new tool is self-apparent, meaning it’s obvious to the testers where and when each new concept will be useful.

Working with developers, any coaching on testing methods will be more successful if demonstrated and proven in the context of the code/feature/product they’re currently working on. I’ve found that I’m not able to update and reuse my existing presentations and materials which are theory-based and use practise testing websites. It’s almost as though the method you’re demonstrating needs to find a bug in your own product during the training session, to be considered a technique worth learning. I picture developers with a small extra toolbox for testing that’s already full, and they’re not convinced yet they need to purchase a larger toolbox!

My own expertise

When doing one-on-one coaching I learn new things every single time, such as domain knowledge, keyboard shortcuts, useful browser extensions, etc. Even so, I’m typically guiding more than I am learning.

While coaching developers it feels more like a two-way learning process. While asking leading questions about quality, scope, and risk, I’m also learning about the operating environment, debugging methods, code structure, in-house test tools, and more. Importantly, I’m seeing how many bugs can be traced back to code patterns and structure, and can therefore be prevented at that level during development. It’s exciting to learn new ways of preventing issues rather than detecting them later.

Which is better?

When I’m working with testers I automatically assume the role of mentor, trainer, leader. For years this has been my comfort zone – testers are my tribe, my people, my community. Personally and professionally, I thrive when I’m just outside my comfort zone. In the past that has lead me to pursue consulting, contracting, senior management roles, public speaking, organising meetups, hosting training courses… Now taking on this new role has allowed me to use my experience in the software industry with quality and coaching, while pushing me to learn about development more deeply.

I’m hopeful that by coaching developers I can have a greater impact on preventing/reducing product quality issues and therefore help to produce quality software faster. It’s still early days.

Please reach out if you’re also in a Quality Coach role and would like to share notes.

Part 1 in a short series of posts about Quality Coaching.

Leave a comment

Posted by on June 13, 2021 in Coaching, Growth, Learning, Quality


Tags: , , , ,

How did you miss that bug?

How did you miss that bug?

That tense feeling we get when users find an important issue in production.
The panic when a manager asks the dreaded question.
Searching through your own notes and test results wondering, how did I miss this…?
Eventually, taking some deep breaths, you remind yourself that exhaustive testing is impossible.

However, you’re not entirely off the hook. As a quality champion for your team you can help to identify risk areas in the software development process and suggest improvements. To bolster your case for change, explain how each improvement could have helped to prevent recent production issues, or at least detect them sooner.

Some recent examples from my own experience include:

  • Test environment was configured differently to the production environment, hiding the issue (i.e. no load balancer).
  • The telltale sign in the log files was missed during testing due to the amount of noise in the error logs and lack of monitoring tools.
  • Test data was too far removed from production data, so the problem wasn’t apparent.
  • Automated test coverage was lacking in some areas, due to limitations in the framework.
  • Some important legacy code was lacking unit tests, which I hadn’t realised.

It’s important that you don’t come across as making excuses, or passing on blame to others. Instead, each point can be used to drive improvements. For ideas on further improvements without waiting for production issues to occur, read The Science of DevOps: Accelerate by Forsgren, Humble and Kim. This book contains excellent ideas for improvements to your team’s processes, backed by data demonstrating links to increases in business performance.

Exploratory testing can help to find issues before they go to production, particularly with edge case scenarios. Exploratory testing is a skill that needs to be learned and practised. There are cheat sheets, heuristics and TestSphere cards to help you find important bugs fast.

Finally, and perhaps most controversially, how much does it matter if some bugs make it through to production? Consider the potential risks of failure; the effectiveness of monitoring and alerting tools used in production; the time required to fix production issues; whether a phased rollout is possible… All of these factors should affect the overall development, testing and deployment approach.

Asking how the testers missed a bug is simply not good enough. Expect more from your managers and your agile team members. In a healthy work environment, the team may ask instead, “How can we all help to prevent this type of issue from occurring in the future?”

1 Comment

Posted by on February 29, 2020 in DevOps, Software Testing


Are You Adding Value, Always?

How do you add value for your company, your team, your customers, and product quality?

The exercises below will help you to focus on adding value through your work, and to confidently describe your contribution to stakeholders.

Analyse and Evaluate

For two weeks, track the tasks you’re working on each day. How is the bulk of your time being spent? The answers may surprise you and your team.

When performing this exercise for myself or with a team member, I keep track of tasks in a mind map as shown below (created using Mindmup). You could also use a spreadsheet.


Figure 1. Example mind map for tracking tester activities

It’s incredibly helpful to review this with your manager. Together you can develop a plan to complete high-value tasks more efficiently, and make low-value tasks obsolete.

To help evaluate the value of each task, consider the following:

  1. Why did you do it?
  2. How did it contribute to your goal of delivering high-quality software?
  3. Is this task genuinely necessary? What is the cost of doing this versus the cost of not doing it?
  4. Could the same outcome be achieved more efficiently?
  5. Could a better outcome be achieved in another way?

Be Efficient and Effective

Next, map each task that you’ve identified to the Cost-Benefit diagram below. 


Figure 2. Consider the relative cost and benefit of tasks

For example, if you’re producing lengthy documents and can confirm that nobody is reading them – stop creating them. Or, if stakeholders are reading only certain sections of your documentation, then those are the only sections you should be producing.

If another task is relatively high cost and high benefit, look for ways to improve efficiency. For example, script repetitive tasks such as configuring test environments, creating test data, or monitoring log files for errors.

To further improve my own effectiveness at work, I try to minimise distractions for a period of time each day. I close email and chat programmes for at least one-hour twice per day, and focus on my other work. Otherwise, I find it difficult to add value in the midst of constant distractions and context-switching. It’s good to experiment and find a time-management method which works for you.

Opportunity Cost

Consider the opportunity cost of activities being performed – by doing any task, you are giving up the opportunity to perform other tasks during that time. Let’s use an example of a manual regression testing cycle which takes three hours to complete. This means having three hours less to spend on activities such as testing user stories, reviewing customer analytics data, or pairing with a developer.  

As professionals, we have a duty to understand the value and opportunity cost of our activities. As an example, manual regression testing cycles are often a lot longer than necessary, and it pays to review them regularly. When I join an organisation and review existing manual regression testing processes, it’s clear that high-impact issues are found during the first 10% of the testing time.

“If you always do what you’ve always done, you always get what you’ve always gotten” – Jessie Potter¹


It’s important to be on the same page as stakeholders regarding the relative cost and benefit of regular tasks.

If you’re required to complete a low-benefit task, perhaps you’ve misunderstood why that particular test activity or document is needed. Ask more questions about the purpose, benefit, and value of the requested deliverables.

If a task seems very expensive, perhaps the cost of doing that task hasn’t been clearly communicated and understood by your stakeholders. Provide high-level information about the steps involved and the time required.

Do you provide metrics which require considerable manual effort to collate? Stakeholders may assume that these figures are readily available.

Do you copy and reuse test documentation and make only minimal changes to the new version? If so, there’s a good chance that nobody is reading these low-value test artifacts, and they’re probably not needed at all.

Work together with your stakeholders to achieve the best outcome. Try to reach a compromise so that you’re only producing deliverables which are valuable, while still providing all of the information required. Then produce those deliverables as efficiently as possible.


As a test professional, it’s part of your responsibility to speak up when asked to do something unproductive. However, pick your battles wisely and start with the highest cost and lowest benefit tasks first. Always offer potential solutions to stakeholders and team members where possible, not just complaints.

It’s important to have a clear understanding of what your manager and your team expects of you. Demonstrate how much of your time is being spent on those goals versus distractions or busy-work with little benefit. Stakeholders understand Return-on-investment (ROI) and cost-benefit analysis terminology. Outline the ROI of planned improvements to testing processes, to help gain support for innovation and change.

The majority of test professionals in Australia and New Zealand are fortunate to have some influence over their own test approach. However, testers working in a less-agile/ more-bureaucratic organisation may have less opportunity to improve their way of working.   

Learning new skills is one way of adding value when you have limited influence. You can learn in-depth about the tools you’re using daily, and discover features which can save you time. Another suggestion is to read online about new test techniques. Remain focused on delivering value to your company and your customers.

Add Value – Always!

The relative cost and benefit of a particular task will change over time. It’s useful to revisit this exercise regularly, e.g. quarterly. The expected outcome is that you’ll be working on the right tasks at the right time, in an efficient and effective way. You’ll be confident in describing your role and the value you add to the team.

After all, if you can’t articulate the value you’re adding, then how will others know?  




Posted by on September 11, 2018 in Software Testing


The Siren Call of Test Metrics

The Siren Call of Test Metrics

This article was originally published in Testing Circus.

Reporting on software testing using metrics can have unintended consequences. Which test metrics do we regularly produce for reporting, and what information will stakeholders interpret from those metrics? Let’s examine some real-world examples in detail.


95% Tests Pass, 1% Fail, 4% Untested

Test pass/fail metrics are popular in traditional test progress reports. Consider for a moment that you are the product owner and you have final responsibility for deciding whether to release a product. The test manager presents you with this graph:

Test Progress chart showing 95% pass, 1% fail and 4% untested
There’s certainly a lot of green in this graph, and green is usually good. So, does this graph make you feel more inclined to release the product?

When I read this graph I have more questions than answers. For example:
Failed tests
– Which tests have failed?
– What was the cause of failure and what’s the severity and status of the related bugs?

Untested tests
– Which tests have not yet been tested?
– What’s the risk of not doing those tests?

Passed tests
How well have the major risk areas been tested?
How were those major risk areas identified?

– What else has not been tested, aside from whatever is represented by the graph, that the test team feels should have been tested? Examples include user scenarios, end-to-end tests, error-handling, performance, security testing and so on.
– What is the test team’s confidence level in product quality?

While the above graph looks more promising than another showing 50% pass and 50% untested, it’s still only a small part of the overall testing story. When viewed in isolation this graph is virtually meaningless, and can provide stakeholders with a false sense of confidence in product quality.

100% Requirements Coverage

This metric sounds reassuring, and looks impressive when presented as a pie chart.

Requirements coverage pie chart showing 100% green

What information is represented here? The data source for this metric is usually a requirements traceability matrix (RTM). A set of tests are compared to a business requirements document (BRD), and every requirement is ‘covered’ by at least one test. The RTM does not – and cannot – take implicit requirements into account, it is limited to explicitly stated requirements.

Some teams report 100% requirements coverage without ever having executed a test, as the RTM matches tests to requirements irrespective of whether those tests have been performed. A team on a traditional project who have painstakingly written tests for each requirement – without ever having seen working software – may report 100% requirements coverage based on the BRD.

During the development and testing cycle, features invariably start to differ from the original requirements. This may be due to implementation limitations, de-scoping of features, changes to the design, and so on. Requirements and written tests quickly become out-of-date. At that point, how useful is this metric?

On a hypothetical highly-organised and well-documented project, if the requirements and tests are constantly kept up-to-date and we perform all of the planned tests, then can we report on 100% requirements coverage? The RTM shows which requirements the testers consider to be covered by at least one test case. It doesn’t show the extent to which the requirement will be tested. As we see in this tutorial from Cem Kaner, 100% test coverage of the system is impossible to achieve in any realistic software testing scenario. We cannot test every input, every combination of inputs, every path through the code.

If you report on this metric do so with caution, and with multiple disclaimers included in your report. Again, this metric can provide a false sense of confidence to stakeholders.

12 Bugs Raised Today

The number of bugs raised per day metric can be enhanced by providing more details, such as a breakdown by severity and trends over time. The major risk in simply reporting this data to stakeholders in a graph is that it may be interpreted in any number of ways.

Graph showing bugs raised each day grouped by severity

Here are some real-world interpretations of this metric from different stakeholders:

  • Developers aren’t testing their own work enough, “Testers are finding too many bugs”
  • Testers should have found these bugs sooner, “Why was a Sev1 only found yesterday?”
  • Testers are working hard at the moment, “They’re raising lots of issues”
  • Testers aren’t working hard enough, “They could be raising at least 20 bugs every day”
  • Product quality is poor, “Look how many bugs were raised in the last two weeks”
  • Product quality is improving, “There’s less bugs now”

Some of these interpretations may coincidentally be true, but could just as easily be incorrect. If your project stakeholders are presented with a graph similar to this one, are they likely to ask for more information, or leap to their own conclusions?

If testers notice stakeholders equating high bug volume to increased testing effort, how might that affect their behaviour? For example, if a tester in this situation found three spelling errors on one page, they might decide to raise them all individually to increase the volume of bugs raised that day. Then there would be an increase in overhead to process those through the bug tracking system. This in turn could be seen as wasting other people’s time, and fracture relationships between the developers and testers.

0 x Sev1, 3 x Sev2 Bugs Outstanding

There are no showstopper Severity 1 bugs waiting to be fixed for this release, and only three critical Severity 2 bugs.

This metric is a great conversation starter, leading to the obvious question: What are the three Sev2 bugs and do they need to be addressed before the product can be released?

There are some less obvious questions to consider:

  • Has the severity of the bugs been reviewed by more than one person?
  • Are these bugs regressions from the current released version?
  • Have the high-risk areas of the product been tested?
  • What other bugs are outstanding?
  • Were any Sev2 bugs downgraded to Sev3 for the purpose of producing this metric?

Focusing only on the high severity bugs may mask high volumes of minor bugs. If there are a group of minor bugs relating to a particular feature or aspect of the product, they can amount to the equivalent of a critical issue. For example, a number of minor bugs in the back-end UI caused staff to spend longer on tasks which they performed frequently, slowing down overall productivity and causing frustration with the system.

Misuse of Metrics

For the most part, reporting on these metrics alone is the sign of an inexperienced, frustrated or unempowered test lead, and is done with good intentions. I know test leads who produce the metrics requested of them (usually by project managers) while feeling frustrated that the metrics don’t accurately reflect their opinion of product quality, because metrics only tell part of the whole story.

A real-world example of this is a project which had at least one test per written requirement, 95% of tests were passing, and there were no Sev1 bugs outstanding. Based on those metrics, the project manager was satisfied that the exit criteria had been met, and that no further time was needed for testing. As we’ve seen above, each of those metrics is flawed if taken in isolation. In this case, the test manager had an uphill battle explaining why further testing time was required to test the major risk areas more thoroughly, while all metrics graphs were essentially reporting a green light.

I have seen a case of metrics being produced which were deliberately misleading. On one project I joined, the previous test manager had asked the team to prepare two test cases per business requirement: one positive test case and one negative test case. Here’s an example of a written requirement from that project:

Staff access to the system is limited according to their role.

It’s entirely possible to write two statements which appear to cover this requirement:

  1. Verify that staff users can access the system functions they should be able to perform 
  2. Verify that staff users can’t access system functions which they shouldn’t be able to perform

While the above requirement is clearly lacking in detail, such as which functions the users should and shouldn’t be able to perform, it’s a real-world example. (The BRD had even been reviewed and signed-off by project stakeholders.) Rather than raising a red flag, the test manager had prepared test artefacts based on the limited information available at the time, and had traced those back to the requirements. This masked the fact that more information was needed before this requirement could be tested.

The test manager was reporting 100% requirements coverage before test execution had begun, based on the RTM. Having now read just one requirement and the associated coverage of that requirement, how much value would you place on that metric? What do you anticipate will happen when the test team attempt to verify whether this requirement has been met by the system?

Moral of the Story

Test metrics and graphs can enhance the testing story, but cannot replace it. Do not present or judge testing progress by test metrics alone, use qualitative information as well. Speak with the testing team regularly to learn the whole story.

These graphs are like the siren songs of Greek mythology. They are appealing, hollow, and will lead to a great deal of sadness and destruction if followed unquestioningly. For alternative reporting methods, see the Reporting on Test Progress section of my article in the Testing Trapeze 2015 August Edition.


Posted by on February 4, 2018 in Software Testing


Tags: ,

What Does Testing Look Like Without Test Cases?

This article was originally published in Testing Trapeze.

At testing meetups, workshops and conferences, context-driven testers are encouraging others to move away from relying solely on test cases. We reason that the prescribed use of test cases and test metrics is an inadequate approach to software testing. We believe that responsible testers must constantly adapt their approach to meet the current needs of the project.

After these events I hear feedback from attendees who are inspired, yet don’t feel that they could move away from using test cases on their project. Some of them anticipate objections from management, and struggle to come up with responses to explain this different testing approach with confidence. The objective of this article is to offer insight into performing software testing and progress reporting effectively, without relying on test cases, and to help you sell and defend this approach to your stakeholders.

Why not use test cases?
As a contract test specialist I’ve had the opportunity to work on a wide variety of projects. Sometimes I join at the start of a project, but more often I’m hired as the hands-on test manager of an in-flight project, with existing test processes, test cases, and a tight deadline.

All of the test cases I’ve ever inherited have been out-of-date – without exception. They were initially written based on requirements documents. As the project details changed, the time\effort needed to keep test cases updated was better spent on testing new features and bug fixes as they were developed. You might have noticed that your own test cases are out-of-date and time-consuming to maintain. Perhaps you’ve wondered if the time you spend re-writing test case steps would be better spent on other testing activities.

At their best, test cases state the obvious to testers who are already familiar with the product. At worst, test cases contain incorrect or obsolete details which lead new testers astray, and require updates which slow down experienced testers like the proverbial ball-and-chain. Let’s look at three common test activities, without the use of test cases: Test preparation, execution and reporting.

Test preparation
As with traditional projects, I map out a test strategy and/or test plan. What’s different is my approach to them. Instead of filling in templates with standard details common to all projects, I create a custom document or mind map to suit the project, using the Heuristic Test Strategy Model (HTSM) as a guide. I constantly reflect on my actions, and ask myself whether the document\section\paragraph is adding value to the project, and whether it is useful and necessary.

So where’s the harm in writing test cases during the test planning phase? The harm isn’t necessarily at the moment of creation; the cost of creating a test artefact also includes the cost of maintenance. Without maintenance, test cases quickly become outdated. Consider whether the information is already available somewhere else, for example, in a requirements document or user stories. It’s more efficient to keep information updated in a single location, rather than duplicating the information across artefacts and across teams.

Instead of spending substantial amounts of time creating difficult-to-maintain test cases, spend this time on other preparatory activities. Read all available documentation, get familiar with existing company products, explore earlier versions of the product and competitor offerings. Improve your product domain knowledge, learn about business processes – present and planned. Talk to developers about the technology stack. Talk to BAs about their design decisions. As you learn, jot down test ideas. This could be in the form of a mind map, checklists, test charters… anything lightweight which can be easily revised.

Test Execution
Working on more-or-less agile projects, I’m focused on testing new features as they’re developed, and keeping up with the pace of development. I’ve found that there’s never a shortage of things to be tested.

My main priorities while testing are to verify that explicit and implicit requirements are met, find bugs and report them. To do this effectively I use a form of structured exploratory testing called thread-based test management (TBTM). This is similar to session-based test management, but testing activities follow a set of ideas rather than being time-boxed. The majority of test ideas are created while testing. I have a Microsoft OneNote file per testing thread, with a new tab each time I retest a feature. Most of my testing threads are feature-based, corresponding to user stories.

When retesting bug fixes, I use a micro-TBTM approach. I treat each bug as a separate thread and enter my test ideas and results directly into the bug tracking tool as a comment. Anyone tracking that bug will be notified of my change, and can provide feedback or ask questions.

For example:

Passed testing in build 33, test environment 1.

  • Postcode field is now populated automatically when suburb is selected – Pass
  • Postcode cannot be manually entered or edited – Pass
  • Still unable to proceed without entering a valid suburb – Pass
  • Typo in validation message, raised separately Bug 2323
  • Suburbs which are duplicated in other states “Silverdale” – Pass
  • Suburbs containing a space in their name “Cockle Bay” – Pass
  • Suburbs which have more than one postcode “Hamilton, NZ” – Pass
  • AU/NZ suburb names containing non-alpha characters – Couldn’t easily locate any, not tested
  • Tested in latest versions of Chrome, Firefox, IE and Safari
  • Tested on desktop and mobile site
  • Updated automated check for customer sign-up

TBTM is traceable, auditable and efficient. It’s great to also have [heuristics] checklists to compare test coverage with, particularly if you start running out of ideas for things to test. I find it helpful to run through my checklists prior to release, to gain confidence in my test coverage.

Reporting on test progress
A lot of my test reporting is verbal, for example, during daily standup meetings and in response to direct questions. I provide weekly test progress reports to stakeholders to reiterate these verbal reports, starting from the initial test preparation stages. The information in these weekly reports feeds into the test summary report at the end of the project.

I’ve learned a powerful secret in my quest to stop reporting on test-case metrics: Senior management appreciate bullet points almost as much as graphs! So I keep my test reports short, useful and relevant.

Below is an example weekly test status report, which lists up to five points each for the most important outstanding product issues, risks, tasks completed this week, obstacles\blockers to testing, and objectives for next week. Consider which headings you would add or remove for your project:

Top issues:
Registration failing in Firefox – Bug 1234
User remains logged in after performing logout – Bug 1235
– Unable to renew accounts before the account expiry date – Bug 1236

Performance test tool license expires next week, payment is still pending approval

Achieved this week:
– Focused on testing new features being developed: registration, login\logout and account page
– Briefly tested renewals and user profiles: no critical issues raised
– Performed thorough cross-browser and cross-device testing of front-end UI

– 2 testers were sick this week
– Test environment data refresh delayed until next week

Objectives for next week:
– Continue testing newly developed features and fixes
– Review performance test plan

Any of these points can be a conversation starter, providing you with the opportunity to tell the testing story in more detail. Given the opportunity, you’ll be able to discuss coverage and depth of testing, and perceived product quality. Don’t feel constrained by the questions you’re being asked. For example:

Project manager: What percentage of test cases has passed?
Test lead: That’s not useful information. Here’s what you need to know…

Adding some metrics to your reports is a valid option. For example the ‘number of resolved bugs waiting to be retested’ or the ‘number of bugs raised this week by severity’. However, be aware when including metrics that you run the risk of readers misinterpreting the figures, or placing more importance on the metrics than the rest of the report.

I like to present test coverage and quality perceptions to stakeholders in a visual model, for example as a heat map. Heat maps can be created with mind mapping software but recently I’ve switched to using tables, primarily because stakeholders are more familiar with using spreadsheets and don’t require additional software to be installed on their computers. Here’s an example:

heatmap table

In this example, the green cells represent a working feature, red indicates feature not working, orange is a feature that works but has issues, grey is untested and white is unsupported. So, at a glance this heat map shows:

  • which areas of the product have and haven’t been tested;
  • a list of supported features per platform;
  • test team impression of quality per feature; and
  • the corresponding issues or bugs.

Introducing context-driven testing
At this point you may have noticed something interesting: all of the above methods can be used alongside test cases. Sometimes this is how I introduce context-driven testing to a team; gradually. Each time, my team has found that as the project progresses, there is less and less value in referring back to their old test cases.

A few weeks after making the switch to context-driven testing on one such project, some managers became nervous that we might be missing something by not executing the test cases. The other testers and I started to wonder if they might be right, so we tried an experiment. Half of the team spent the next three days executing test cases, and the other half of the team continued with structured exploratory testing. We met at the end of the three days to review our findings.

Even I was surprised to find that the testers executing test cases had not found a single new bug. They felt frustrated and constrained by the outdated test cases, and had put pressure on themselves to get through the test cases as fast as possible. A large chunk of their time was spent looking up existing bugs they had raised, in order to add the issue number reference to their failed test case results. We also realised that they had hardly spoken to a developer or business representative.

Less surprisingly, the half of the team performing structured exploratory testing found the exact opposite. They had continued to work closely with the project team, raising bugs and progressing with feature testing coverage. After that we agreed to leave the test cases in favour of using the business requirements document as a checklist, to confirm whether we’d missed testing any important product features.

There is no one-size-fits-all “best-practice” approach to software testing. I have implemented a new context-driven testing approach for each project I’ve worked on in the last 18 months, across different teams, companies and industries. I’ve earned the respect of project managers, testers, developers and business stakeholders. The turning point for me came after Let’s Test Oz 2014, when I took Fiona Charles’ words to heart – “I’m not going to do bad work”.

The first time I refused to use existing test cases I was nervous. I knew the principles behind context-driven testing and I was keen to implement change. I also felt that I was going out on a limb and putting my professional reputation on the line. Yet it was surprisingly easy to demonstrate the value of context-driven testing approaches to stakeholders, and to gain their support.

Convincing teams to adopt context-driven testing approaches became easier over time. As I built up a portfolio of success stories my confidence improved. I don’t expect junior testers to have this capacity for change within existing project teams. The responsibility lies with test managers and test leads to demonstrate to project stakeholders what software testing really involves.

Context-driven testing is not easy and it’s not boring. It’s challenging, rewarding and – best of all – it’s a more effective way to test. Testers following these principles are more motivated, engaged, valued and respected.

Remember that testing is about providing valuable information, not just the percentage of passed test cases.

With thanks for inspiration:


Posted by on June 29, 2017 in Learning, Software Testing


One Tester’s Account of an iOS Update

One Tester’s Account of an iOS Update

As a tester I’m well aware that things can go wrong during software updates. I always wait at least a week before upgrading iOS on my iPhone, leaving time for issues to be reported and fixed. When I run the upgrade, I’m expecting something to go wrong. This means that I always backup my phone first – luckily!


During the latest upgrade, I saw this on my iPhone:



I’d been expecting something to wrong and my iPhone cable is looking a bit worse-for-wear these days, so I assumed that my phone had been disconnected. Doesn’t that screen look like it’s asking me to plug in my phone?


Naturally I unplugged and plugged in the cable again to restore the connection.


In hindsight I think that image means “iTunes is updating your phone, do not under any circumstances unplug the cable”. From a usability perspective, this image is ambiguous at best. Poor usability combined with my pessimism caused the update to fail.


I’ve had only one iOS update fail in the past, around 2 years ago, yet since then I pretty much expect them all to fail. A catastrophic failure can leave a lasting impression of poor software quality.


A popup dialog said that I needed to run the Restore next which would clear all of my phone settings and restore from backup. That sounded straightforward to me, because I’ve restored my phone in the past with no issues (due to hardware upgrades and replacements). Based only on my personal experience, my impression was that the Restore feature is more reliable than the Upgrade feature. In fact, the upgrade has failed for me just once out of many uses, while the restore has always been successful but I’ve only used it four times.

Then the restore from backup failed, with Unknown error 21.



The dialog with the unhelpful wording led to a website with clear instructions. Some steps were less helpful than others, e.g. Upgrade Windows and reboot your PC.


After following the steps I tried again, and this time the Restore process looked like it was working.


Then iTunes asked for my backup password. I’d just created a backup this morning and hadn’t been asked to give it a password…? I tried my Apple ID password, with no luck.


I found an Apple forum post where someone was complaining about having no way to recover the password. The Apple fan club had gone the offensive, “How could you forget your password? This is your own fault!”. The defence was, “How can I be expected to remember my password from years ago when I bought my first iPhone?”


Aha! A clue… I tried one of my old passwords from days gone by and it worked! The person complaining was more help to me than the Apple-defenders.


My iPhone is now restoring from backup and I’m breathing a sigh of relief. I feel like it could be smooth sailing from here, and there’s only 22 mins remaining.

Now 27 minutes.

Now 34 minutes.



Leave a comment

Posted by on July 30, 2016 in Software Testing


WeTestAKL Meetup 99-sec Talks

Our July WeTest Auckland Meetup came with this warning:

If you attend this meetup, you will be speaking!

Inspired by TestBash, Shirley Tricker organised a meetup where everyone was required to speak.. but only for 99 seconds!

We started with two 25-minute talks from first-time speakers, on mobile test automation and communication skills for testers. Then we launched into the 99-sec talks.

Shirley chose four people at a time to come to the front and speak for 99 seconds each, to a group of 16 testers. Some people had clearly practiced their timings, delivering their closing sentences just seconds before the buzzer. Many more were surprised when the buzzer sounded mid-sentence!

I’m so impressed that everyone who came along spoke: including people who’d made it off the waiting list just one hour before the meetup; some first-time attendees; plenty of first-time speakers; and those who hadn’t read the Meetup properly and didn’t realise that they needed to talk!


Lauren Young- What is a Statement of Work?

Kasturi Rajaram- Why I love Non-Functional Testing

Rasha Taher – HiPPO: Highly Paid Person’s Opinion

Sandy Lochhead- Child’s Play, the Value of Playing Games

Nadiia Syvakivska – Luck

James Espie – Maintaining Perspective

Reshma Mashere – Listening Skills and Meditation

Monika Vishwakarma – Thank You WeTest

Bede Ngaruko – Fear of Failure

Jennifer Haywood – Mobile Testing Mnemonics

Inderdip Vraich – Specflow

Nav Brar – My Journey to NZ

Shirley Tricker – Career Tips

Laurence Burrows – Raising Defects

Jennifer Hurrell – Documenting Exploratory Testing

Kim Engel – #30DaysOfTesting Challenge

Thank you Shirley for creating such a friendly and fun environment where we could share and learn. And thank you to our sponsors for the evening, Assurity.

* More details on the 99-sec format:
1 Comment

Posted by on July 30, 2016 in Software Testing


As a software tester, do I need to learn about automation?

Testers regularly ask, “Do I need to learn automation skills?”.

Let’s step back for a minute and put this another way, “Do I need to learn how to drive a car?”.

Well you could walk, or take the bus, or pay a taxi\Uber driver to drive you around. These are all valid choices. But there’s a big advantage in being able to drive, for times when driving is the best option.

So my answer is no, you don’t need to learn automation skills, but having those skills will let you make informed decisions about the most efficient way to approach each testing task, with a wider range of options available to you.

Consider for a moment, what has stopped you from learning hands-on automation skills before now?


Ignore the Fear

When testers ask me whether they’ll need to learn automation skills, there’s an implied reluctance or fear behind the question. Sometimes it’s a fear of being left behind in the job market, or of being replaced by testers with stronger technical skills. In a few cases the testers asking seemed reluctant to invest time in learning new skills unnecessarily. I have a lot more time for the first group of people, compared to the second group.

For the testers who ask this question, are you nervous, intimidated or overwhelmed when it comes to learning automation skills to help with software testing? For now, let’s pretend that you’re not. Imagine instead that you’re completely capable of writing an automation script (because you are). How can you get started?

Avoid Analysis Paralysis

There is so much information online that we’re presented with an overwhelming number of options and learning resources. I got stuck while deciding which programming language to learn (COBOL has gone out of fashion since my development days).

I had finally decided on Python – I can’t remember why now – and had completed the first few tutorials, when others convinced me I should be learning Java. Or C#, or Ruby… So I stopped doing the Python tutorials, and unfortunately it was a while before I decided to try again.

I’ve since learned that it simply doesn’t matter. My learning experience has driven my approach for this post. The key thing to remember is that once you learn the basics of an object-oriented programming language, and an automation tool, it gets much easier to learn the next one. They all have a lot in common.

To help you stop procrastinating, and to make sense of all the options out there, the rest of this post is very straightforward. My goal is to get testers to learn some automation, to complement their existing skill sets.


Take the First Step

Learn Java, using this website to get you started:

(The time you may be tempted to spend on researching the best website\tutorial to use could be spent learning Java instead. )

You know your own preferred learning style better than I do, I just have a few words of general advice.

  1. Start from the beginning.
  2. Follow the instructions to set up your computer, and try to overcome any perceived obstacles while you’re getting started.

i.e. if you get stuck, ask for help.

  1. You won’t need to memorise anything. Yes, the maximum “int” value is 2,147,483,647. And no, you won’t need to remember that number. Once you’ve read the course information, you’ll know where to find it later if you need it again.
  2. It’s okay if some things don’t make sense right away. “Float data type is a single-precision 32-bit IEEE 754 floating point”. I don’t understand most of that sentence, yet I can write automation scripts.

 Java tutorial.JPG

You don’t need to complete all of the lessons unless you want to. You’re not trying to become a developer, you need to know ‘just enough’ to be getting on with for now. The course material will still be there when you need to deepen your knowledge of Java, down the track.

There are no stupid questions. Anything you want to ask has probably already been asked and answered on If you can’t find an answer there please feel free to post your question. You’ll be doing a huge favour to plenty of other beginners like yourself, who will benefit from reading the responses in the future.

Take the Next Step

So, what’s next?

Pick a tool, any tool.. If your test team aren’t currently using any automation tools and you’re stuck for ideas, check out Ghost Inspector There’s a free trial and you can quickly automate some website checks. Ghost Inspector is like Selenium, but it’s hosted on the cloud with a pretty good GUI, and a record and playback feature, making it easy to get started with.

ghost inspector

Start by using a non-production version of the website you’re currently testing, or use a sample website created specifically for learning to use automation tools. For example,

The first test scripts you create will fall into all the traps inherent with using record and playback tools, and that’s okay! Because you’ll be automating, and you’ll be learning, and that’s an excellent start.

From here you’ll gain confidence to start\join a conversation about automation, you’ll have specific questions to ask, you’ll be able to research online independently, and continue to learn new automation skills.

 demo site.JPG

It’s Okay to Fail

They say in Silicon Valley, “Fail fast, fail often”. The same applies here.

Learn the hard way by diving in, getting started, making mistakes, and improving as you go. The worst thing you could do is read every “Lesson learned in test automation” article on the internet before learning to use a single automation tool.

Start Right Now

Get started with an online Java course right now –

If you prefer hands-on classroom-style learning, at Engel Consulting we have Introductory Java and Selenium courses available in Auckland, with more courses coming soon. There may be also local training providers in your area, or experienced testers willing to provide training on request.

Also worth a mention is Mike Talks’ blog post series on automation:


Posted by on July 24, 2016 in Software Testing



What Are Test Oracles?

What Are Test Oracles?

How do you recognise that something you’ve seen may be a bug?

What do you do when your test results don’t match the expected results?

For our WeTest Meetup discussion we started with a working definition of an oracle as “a source of authoritative information”.

We rarely rely on just one source of information when determining how a product should work. For example, when working from a list of written requirements we may also check how the previous version of the product works, to further clarify our understanding.

Two authoritative sources can contradict each other. For example, written requirements may not match the design documents. In that case, how do we decide which oracle to use?

Stakeholder decisions during product development can be a compromise between desired functionality and a practical solution. How often is documentation updated to reflect these ever-changing decisions? In this situation, is ‘product team consensus’ considered to be an oracle?

An oracle can be incomplete. In fact, we struggled to think of a real-world example where an oracle would be complete and correct…

Rapid Software Testing (RST) takes a broader definition of an oracle:

“An oracle is a means by which we recognise a problem when it happens during testing” – James Bach and Michael Bolton, Rapid Software Testing.

Rasha Taher brought along this RST diagram on oracles to share with the group:


Our authoritative oracles all fall under the Reference category in this model. They are external, explicit oracles. This model opened up a whole new perspective to the discussion. We started to brainstorm other oracles we use every day, without necessarily realising that we’re using them.

After talking through this model as a group, we felt more conscious of using our own experience and feelings as oracles while testing. Did the behaviour of a certain feature make you feel confused? You may have found a usability issue, for example. Once you’ve worked out the cause of your confusion, consider whether users may encounter the same problem, particularly new users.  Will they be provided with more or less training than you received as a tester? Is there a way for users to overcome this hurdle, without needing to personally ask the developer and product owner how the feature should work?

Conference oracles highlight the importance of communication in software testing. With open and frequent communication testers can gain a clearer picture of stakeholders’ expectations of the product. This can help guide testing, to determine how well the product meets those expectations.

While discussing Inference oracles, Ram Malapati led us to the FEW HICCUPPS heuristic. These are a whole topic by themselves, and could be the main topic for our next discussion group!

Through learning the term ‘test oracles’ and reviewing this model, we feel more empowered and in control of our test approach. While we may not use the term oracles at the office, just being aware of the oracles we use daily can improve both our approach and our confidence.

Finally, learning the term ‘test oracles’ opens up a new avenue of research on software testing methods, to learn more about oracles and how we can use them.

Further reading:
As Expected – Michael Bolton
Oracles from the Inside Out – Michael Bolton
What Testers Find – James Bach

Leave a comment

Posted by on July 24, 2016 in Software Testing

%d bloggers like this: