How do you recognise that something you’ve seen may be a bug?

What do you do when your test results don’t match the expected results?

For our WeTest Meetup discussion we started with a working definition of an oracle as “a source of authoritative information”.

We rarely rely on just one source of information when determining how a product should work. For example, when working from a list of written requirements we may also check how the previous version of the product works, to further clarify our understanding.

Two authoritative sources can contradict each other. For example, written requirements may not match the design documents. In that case, how do we decide which oracle to use?

Stakeholder decisions during product development can be a compromise between desired functionality and a practical solution. How often is documentation updated to reflect these ever-changing decisions? In this situation, is ‘product team consensus’ considered to be an oracle?

An oracle can be incomplete. In fact, we struggled to think of a real-world example where an oracle would be complete and correct…

Rapid Software Testing (RST) takes a broader definition of an oracle:

“An oracle is a means by which we recognise a problem when it happens during testing” – James Bach and Michael Bolton, Rapid Software Testing.

Rasha Taher brought along this RST diagram on oracles to share with the group:

Our authoritative oracles all fall under the Reference category in this model. They are external, explicit oracles. This model opened up a whole new perspective to the discussion. We started to brainstorm other oracles we use every day, without necessarily realising that we’re using them.

After talking through this model as a group, we felt more conscious of using our own experience and feelings as oracles while testing. Did the behaviour of a certain feature make you feel confused? You may have found a usability issue, for example. Once you’ve worked out the cause of your confusion, consider whether users may encounter the same problem, particularly new users.  Will they be provided with more or less training than you received as a tester? Is there a way for users to overcome this hurdle, without needing to personally ask the developer and product owner how the feature should work?

Conference oracles highlight the importance of communication in software testing. With open and frequent communication testers can gain a clearer picture of stakeholders’ expectations of the product. This can help guide testing, to determine how well the product meets those expectations.

While discussing Inference oracles, Ram Malapati led us to the FEW HICCUPPS heuristic. These are a whole topic by themselves, and could be the main topic for our next discussion group!

Through learning the term ‘test oracles’ and reviewing this model, we feel more empowered and in control of our test approach. While we may not use the term oracles at the office, just being aware of the oracles we use daily can improve both our approach and our confidence.

Finally, learning the term ‘test oracles’ opens up a new avenue of research on software testing methods, to learn more about oracles and how we can use them.

Further reading:
As Expected – Michael Bolton
Oracles from the Inside Out – Michael Bolton
What Testers Find – James Bach