I’ve wasted a lot of my time estimating future testing activities down to the number of days, for minor and major releases of complex systems. Now I’m considering a new approach.
I will review whatever materials are available for the project, speak to stakeholders about their primary values and concerns for the project, gauge the budget for number of test analysts and choose one of the following estimates:
– 6 weeks
– 3 months
– 6 months
The numbers may change slightly of course, but essentially this is the level of estimate breakdown that I’m prepared to provide at the outset.
My reasoning is that any up-front estimate will be rendered inaccurate due to a large number of unknown factors, and past experience at detailed estimation efforts have proven to be futile. I can continue wasting my client’s time and money, or we can take a broader view of estimation and revise that estimate when I’m elbow-deep in the project and have a clearer understanding of the risks involved.
A key success-factor for this approach will be transparency and regular reporting. To be effective in my role as test manager, I need the support of the project manager and stakeholders. They are much more likely to be supportive when provided with a clear understanding of the top few goals, achievements, risks and issues facing the team each week.
So, how can I tell if testing is running on-time or going over schedule? Well, I’d have to ask you to consider – what does that question even mean? If your testing is running to plan (somehow) and you find a defect which causes a decent-sized code re-write, is your testing still running on time? Was it in any way useful to report that you were running “on time”? Instead I report clearly on what has been tested, the most important outstanding issues and the product areas which are yet to be tested.
Another key success factor is commitment to quality and efficiency. Only when the whole team are focused on delivering a quality product efficiently can this approach to simplified\iterative estimation be successful. Simplified estimation is not about laziness, it’s about expending efforts where they will be of most benefit.
Thoughts?
Is it good enough yet? 
Geoff Horne
June 9, 2014 at 8:11 AM
Hello Kim
I think it’s OK to be high-level and vague-ish up front however as execution progresses, there is information we can capture on a daily basis eg. how fast test execution is travelling, how many defects we’re finding, how fast they’re being fixed etc (using whichever measurement methods make sense in context) then use this info to forecast ahead. Of course, we can never know what’s just around the corner however by tracking daily, ‘around the corner’ encounters become part of the info captured and then accounted for in further forecasting. I’ve found this approach to be of benefit when I have a programme of projects/workstreams, all travelling at a different pace but required to meet at a common end point.
Geoff
LikeLike
Joe DeMeyer
June 9, 2014 at 10:23 AM
I adamantly support an iterative approach to testing estimation as you suggest. An estimate is information at a point in time and, as you point out, conveys what you know about the work when you made the estimate.
With this approach, you can update your estimate as more information (such as work complexity, defects, extended development) becomes available (perhaps weekly or more frequently if warranted).
You are spot on Kim when you recommend letting the benefit of testing guide the testing effort. In this manner, it becomes a conversation about a testing budget rather than a struggle to meet a date: “Where should we spend our testing hours now?” is much more collaborative than “Can all this testing be finished by this date?” This has been where I’ve been driving these conversations recently.
I also like the “whole team” approach also! I think moving the mindset about testing from a testing team activity to a project team activity engages the project team in quality and should make everyone’s contribution more valuable. I would be interested in sharing thoughts on this.
Great post!
LikeLike
Tony Ascenso
June 15, 2014 at 12:00 PM
Love the approach, Kim. It is all about common sense, looking at today’s situation and leaving room for learnings as project(s) progress. Estimates are exactly that, estimates! You don’t know what you don’t know, especially on large projects and/or where dependencies and inter-dependencies can raise their ugly heads. Only have one issue. It won’t stick in today’s commercial world, especially where we are pinned down to sometimes upfront estimates or/and on fixed price dealings (shaking my head as I have been there recently…). Great post and comments from Joe DeMeyer 2. Cheers
LikeLike
Jim Grey
July 26, 2014 at 4:53 AM
It warms my heart that the first post I see on a test manager’s blog is about estimating.
I like an approach where you estimate roughly but keep re-estimating roughly as the project progresses to re-project delivery. It can lead to ongoing negotiation about risk, coverage, and schedule, so that the right level of quality can be delivered at the right time.
LikeLike
Peter B
July 30, 2014 at 8:56 AM
What?! Are you telling me that I should give up on searching for a magical tool that does my estimation based on some mythical test case count or test point count? I am flabbergasted.
Seriously though, it’s the experience that qualifies us to be Test Managers and to be able to do what you suggest and in my opinion there are far too many people running around with the title and not the qualifications.
The only thing I would add is that when I do the same thing I do have some idea in my head of the breakdown i.e. Planning Vs Preparation Vs Execution so if I am questioned I have something to explain how I arrived at the numbers. Oh, I suggest you add 10% as it is far easier to adjust down that to extend out at a later date.
Great piece.
Regards
LikeLike
Kim Engel
July 30, 2014 at 9:13 AM
Breakdowns are useful for people who don’t (won’t) understand that planning continues throughout execution. And there are many such people, the collective noun being ‘stakeholders’ 🙂
Seriously though, I believe that all stakeholders understand that planning in general is an ongoing activity, they’re all involved in some level of planning themselves. Which makes me wonder where this request for breakdowns, which we see time and time again, is actually coming from. Perhaps they don’t recognise that test planning faces the same challenges as planning in general. Or perhaps MS Project requires a start and end date for such activities.
Hmm, 10%.. So I should estimate 6.6 months? Or ‘From now until Go Live, plus 10%’?
There’s little point in attempting to add specific contingency to vague estimates. Instead I operate transparently, and keep stakeholders informed as much as possible. We’ll see where that gets me. It probably helps that I’m focused on project goals and have no interest in being promoted, because I doubt that my approach is suited to promotions.
LikeLike