Author Archives: Kim Engel

About Kim Engel

Software Test Manager

What Are Test Oracles?

What Are Test Oracles?

How do you recognise that something you’ve seen may be a bug?

What do you do when your test results don’t match the expected results?

For our WeTest Meetup discussion we started with a working definition of an oracle as “a source of authoritative information”.

We rarely rely on just one source of information when determining how a product should work. For example, when working from a list of written requirements we may also check how the previous version of the product works, to further clarify our understanding.

Two authoritative sources can contradict each other. For example, written requirements may not match the design documents. In that case, how do we decide which oracle to use?

Stakeholder decisions during product development can be a compromise between desired functionality and a practical solution. How often is documentation updated to reflect these ever-changing decisions? In this situation, is ‘product team consensus’ considered to be an oracle?

An oracle can be incomplete. In fact, we struggled to think of a real-world example where an oracle would be complete and correct…

Rapid Software Testing (RST) takes a broader definition of an oracle:

“An oracle is a means by which we recognise a problem when it happens during testing” – James Bach and Michael Bolton, Rapid Software Testing.

Rasha Taher brought along this RST diagram on oracles to share with the group:


Our authoritative oracles all fall under the Reference category in this model. They are external, explicit oracles. This model opened up a whole new perspective to the discussion. We started to brainstorm other oracles we use every day, without necessarily realising that we’re using them.

After talking through this model as a group, we felt more conscious of using our own experience and feelings as oracles while testing. Did the behaviour of a certain feature make you feel confused? You may have found a usability issue, for example. Once you’ve worked out the cause of your confusion, consider whether users may encounter the same problem, particularly new users.  Will they be provided with more or less training than you received as a tester? Is there a way for users to overcome this hurdle, without needing to personally ask the developer and product owner how the feature should work?

Conference oracles highlight the importance of communication in software testing. With open and frequent communication testers can gain a clearer picture of stakeholders’ expectations of the product. This can help guide testing, to determine how well the product meets those expectations.

While discussing Inference oracles, Ram Malapati led us to the FEW HICCUPPS heuristic. These are a whole topic by themselves, and could be the main topic for our next discussion group!

Through learning the term ‘test oracles’ and reviewing this model, we feel more empowered and in control of our test approach. While we may not use the term oracles at the office, just being aware of the oracles we use daily can improve both our approach and our confidence.

Finally, learning the term ‘test oracles’ opens up a new avenue of research on software testing methods, to learn more about oracles and how we can use them.

Further reading:
As Expected – Michael Bolton
Oracles from the Inside Out – Michael Bolton
What Testers Find – James Bach

Leave a comment

Posted by on July 24, 2016 in Software Testing


Listen to Your First Testing Podcast

Listen to Your First Testing Podcast

This month I’m joining many members of the online software testing community in the 30 Day Testing Challenge.

The challenge for Day 3 is “Listen to a testing podcast”.

Testers know that rules are made to be broken… I was planning to watch a testing talk on YouTube instead. When my colleague Ram insisted that I should actually listen to a podcast, I had to confess that I’ve never listened to a podcast before.

Here’s what I’ve learned about how to get started. Unsurprisingly really, it’s easy!

I’ve included 3 testing podcast channels below. If you know of other great channels please add them in the comments section.

  1. Search for\open the Podcasts app (see links below for app recommendations)
  2. Search for testing podcast e.g. Ministry of Testing, Testing in the Pub, Test Talks…
  3. Subscribe to the channel
  4. View the channel feed
  5. Download an episode that looks interesting
  6. Listen at any time

This slideshow requires JavaScript.

Podcast Terminology

Podcast – term used to refer to either a channel or an episode (confusing hey..)
Channel – a collection of podcast episodes, which users can subscribe to
Episode – an individual mp3 file, which can be downloaded or streamed
Feed – an updated list of episodes for a podcast channel


There’s lots of other podcast apps available for iOS and Android, with extra features.


Posted by on July 2, 2016 in Software Testing


“7 Testing Principles” – Meetup takeaways

“7 Testing Principles” – Meetup takeaways

Over lunch time today our WeTest meetup group discussion topic was the ‘7 Testing Principles‘.

Our aim was not to dissect and review the principles.. We only had 50 minutes – less the time it takes to order and pay for lunch! Instead we used the principles as a focus point to discuss relevant aspects of our current project contexts and our past experiences.

From principle 3 ‘Early testing’ – my takeaway was that testing involvement is more useful before and after architectural design rather than during architectural design. I think this could be a whole separate topic for discussion\debate, depending on other testers’ experiences and context.

There were also real-world stories of what can go wrong when testing is not done early e.g. finding issues with requirements during UAT, on an “Agile” project!

From principle 6 ‘Testing is context dependent’ – We had fun explaining different aspects of context and how they affect our current projects. My key takeaway is that context can change even within the same team and project. Eg, one company grew from 5 to 150 people, causing a major context shift.

The question was raised, “Can automation be context-driven? “. Again, that will be a good topic for a future discussion. I’d like to discuss this in more depth.

Principle 2 “Exhaustive testing is impossible” – This came up briefly, and basically we all nodded 🙂

At that point we were out of time, and we finished by unexpectedly giving Elena (a Neighbourly developer) some direct user-feedback.

This meetup was interesting, easy to organise through our existing Meetup group, and fun to attend. I encourage you to consider hosting a similar discussion group.

Leave a comment

Posted by on April 29, 2016 in Software Testing


Rapid Software Testing – Reading Recommendations

Having just completed Rapid Software Testing twice in two weeks with James Bach, I’m feeling motivated and inspired to continue learning.

Here’s a list of books recommended by James during the course. These will enhance your skills and change the way you look at testing.


The first book may be the most important, and the most difficult to read. I’m still getting through my copy. The content is excellent, and there’s a lot to take in.
The next 4 books are real page-turners, explaining important and complex information is a way that’s enjoyable to read.
I haven’t yet read the last book on this list.

An Introduction to General Systems Thinking by Gerald Weinberg
Thinking, Fast and Slow by Daniel Kahneman
Tacit and Explicit Knowledge by Harry Collins
Lessons Learned in Software Testing: A Context-Driven Approach by Cem Kaner, James Bach, Bret Pettichord
The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully by Gerald Weinberg
Discussion of the Method: Conducting the Engineer’s Approach to Problem Solving by Billy Koen

If you’ve already read these books, I’m interested to hear your thoughts. For example, what was the biggest takeaway you got from each book, and how has that helped you with software testing?


Posted by on August 14, 2015 in Learning, Software Testing


Tags: , , , ,

What’s Your Context? – Workshop with Fiona Charles

How do you discover the differences in context between clients and projects, and whether those differences matter? While intuition is important, unconscious analysis and choices lead to unconscious assumptions – Fiona Charles

Ask a tester which approach is the best way to test software. The typical response will be “It depends”. But what does it depend on, and why? How will those facors affect testing?

Fiona presented her “What’s Your Context?” workshop to the Auckland WeTest meetup group. We split into 6 groups to brainstorm the elements of context that affect our approach to testing. It’s difficult to report on the value gained from attending a workshop, as the learning comes from being involved in the discussion. Here I’ve recorded my brief notes on the “Aha!” moments described by others at the end of the workshop.

Me: I’ll be doing this exercise regularly for projects I work on. More aware how much context can change during the course of the project.

Natalia: It was very useful to learn about other tester’s contexts.

Pete: Two favourite sayings are ‘It depends’ and ‘Why?’. For example, ‘Why are we doing this?’.

Morris: We say ‘team’ a lot. Testing is a team sport.

Vikas: Highlights the importance of thinking about context deliberately, instead of repeating past processes.

Shaheem: Maintain a risk focus, which things can derail or negatively impact the project.

Chris: The definition of Minimum Viable Product is heavily context-dependent.

Georgia: Focus on the problem being solved. Take history into account. Appreciate the benefit of informal communications (this arose in the context of working with offsite teams).

Vincent: Like the experience of working with his team, learning from their experiences, and the way they grouped elements of context together into personal, product, team and development methodology.

John: So many sources of information beyond just requirements. Stop and consider context first before you get started.

I encourage other testers to take Fiona‘s workshop. Her questions, insights and stories brought the exercise to the next level.
In the meantime consider pairing up with one or more testers and asking yourselves, “Which elements of your context affect your testing? Why, and how? How can you use this information to improve your testing approach for your current project?”

1 Comment

Posted by on November 10, 2014 in Software Testing


Tags: , , ,

Let’s Test Oz – Closing Keynote from Fiona Charles

The closing keynote speech of Let’s Test Oz 2014 was “The Battle for Our Hearts and Minds” by Fiona Charles.

This was the first time I’ve attended one of Fiona’s talks in person. My impressions of Fiona after this conference are that she is honest, practical, a strategic thinker and that she doesn’t mince words.

“I have seen 50-page test strategies without an ounce of strategic thinking”

Fiona Charles

“I’m not going to do bad work” Fiona Charles

The theme of this keynote was that attempts to standardise testing are stifling creativity and value, and that it’s time for testers to take back our craft. Fiona spoke of the need for testers to have the courage and tenacity to speak up about important issues when others remain quiet. This included being willing to ‘blow the whistle‘ where necessary to expose important issues which could affect people’s lives.

“We need to be able to say things that nobody wants to hear, because that’s our job”

The topic of testing standards came up more than once, as a primary cause of the long-term de-skilling of the testing workforce and the current overall state of testing processes and documentation. Using the example of a 25-page IEEE 829 compliant Test Plan, Fiona saw no project-specific content until page 12. The time taken to produce these documents is costing companies money, and contributes to testing being viewed as ‘too expensive’. The focus of testing should be on adding value to the project and to the company.

“The Master Test Plan is probably the most useless document since printing was invented”

Most of the people behind the creation of ISO 29119 stand to profit if the standard is introduced. Interestingly, Fiona’s opposition to ISO 29119 comes despite her anticipation that she’ll profit from the standard if it’s introduced. Fiona described how she has seen first-hand the damage caused by compliance to the IEEE 829 test documentation standard. She has been called in to multiple organisations to mop up the damage which that standard leaves in its wake, and she has every reason to believe that ISO 29119 would create more of the same damage.

“The quest for certainty collides with the reality of software development”

Fiona introduced the concept of “healthy uncertainty vs unhealthy certainty” while debunking the notion that popular test metrics are useful. She covered some key attributes of great testers, and they’re not the ones you see listed in jobs ads: Integrity, Independence of Mind, Courage, Engagement…

I really enjoyed this talk. It was motivational, inspirational and a call to action for all testers.

Recommended reading\viewing:
The slides from this keynote are available from the Let’s Test Oz website.
Breaking the Tyranny of Form blog post – Fiona Charles
Delivering Unwelcome Messages EuroSTAR webinar – Fiona Charles
Slides from We are the 99% – Anne-Marie Charrett

All quotes in this post are from Fiona Charles’ keynote.


Posted by on October 10, 2014 in Software Testing


Tags: , , , ,

Equality at IT conferences

At the Let’s Test Oz conference I actually forgot about gender inequality in IT for three days.

I don’t complain about gender inequality in the workplace per se, for the usual reasons which I try not to complain in general. It’s not efficient, it’s not effective, and no-one listens anyway… I prefer to act. For example:
– Supporting and advocating for my team members who’ve escalated issues of harassment (albeit to no avail).
– Working to ensure that team members can return to work part-time after a career break if required.
– Championing pay rises for team members who are comparatively underpaid (and usually aren’t aware that they’re underpaid).

At most IT events I can’t help but notice that I’m in the minority. At CITCON 2014 approximately 10% of attendees were female. At a recent Splunk seminar in Auckland less than 5% of us were female, and that was confronting. On the other hand I’ve gotten used to management meetings with predominately male co-workers, because that’s the norm today.

The Let’s Test Oz 2014 conference was an exception. During the conference wrap-up Anne-Marie Charrett observed an almost equal number of men and women at the conference, both speakers and attendees. I looked around and saw that she was right, and then I allowed this to really sink in… The gender distribution at the conference matched the real world closely enough that I’d gotten through an entire IT conference without noticing the percentage of women attending. That’s progress!

While writing this I realised that I’ve never noticed gender at the Auckland Testers Meetups either, again because attendance is representative of the real-world. I think it’s great to notice and celebrate these examples of progress.

I’ve written for the Women Testers magazine and attended the Women in Tech meetup to support both initiatives, to further my learning, and to build my professional network and industry profile. I was silently cheering for the few men at the last Women in Tech meetup because they genuinely came to listen and learn about the issues women face. We could do with more of that in tech.

Leave a comment

Posted by on October 10, 2014 in Equality


Tags: , , ,

%d bloggers like this: