The Estimation Trap: Why Software Projects Keep Missing the Mark

Here’s what every tech lead needs to know about the reality of software estimations - and why we might be doing it all wrong.

The Hard Truth About Estimations

From the Chaos and Gartner reports, the numbers are stark:

• 80% of projects fail or miss deadlines
• Only 16.2% of projects finish on time and on budget
• Average overruns: 63% time, 45% cost
• 17% of large projects threaten company existence

The Estimate Paradox: When Guesses Become Shackles

We can give for granted that estimations are based on guesses, and they are always wrong. The worst consequence of an estimation is that they bound you into a mental contract, you estimate 5 days and everyone around will expect to be ready in 5 days. This can lead to different side effects that you don’t want to have when dealing with a project.

• Forces rushed solutions over quality code
• Ignores emerging complexities
• Overtime Culture

Expecting or missing estimated deadlines can cause frictions within the team, toxic competitions are evil.

What is the numerical impact of this?

Research shows:

• 71% of developers report stress from estimation-based deadlines
• Teams with strict estimation adherence show 3x more burnout
• Quality drops 45% when teams rush to meet estimates

The Story Behind Story Points: A Short Circuit in Agile Estimation

In the realm of Agile development, story points have become a ubiquitous tool for estimating the complexity of tasks. However, their origin and application reveal an intriguing paradox: a method designed to abstract time has inadvertently circled back to it. Let’s delve into the story behind story points and explore their role in Agile estimation.

The Origin of Story Points

Story points were introduced as a way to estimate the complexity and effort required to complete a task, without directly tying it to time. The idea was to focus on the relative difficulty of tasks, considering factors like risk, uncertainty, and effort, rather than just the hours or days needed.

The Short Circuit: Complexity to Time

Despite their intention to abstract time, story points often end up being converted back into time measures. This short circuit occurs when teams use story points to calculate velocity and predict timelines, effectively reintroducing the time element they were meant to obscure.

How it happens:

1. Complexity Estimation: Teams assign story points based on perceived complexity. 2. Velocity Calculation: The number of story points completed per iteration is tracked. 3. Time Prediction: Velocity is used to estimate how long future tasks will take, bringing time back into the equation.

The Paradox of Story Points

This cycle creates a paradox where story points, intended to provide a more nuanced view of work, end up serving as a proxy for time estimation. This can lead to several issues:

• Misalignment: Teams may focus on hitting point targets rather than delivering value.
• Pressure to Inflate Estimates: To meet deadlines, teams might assign higher points to tasks.
• Loss of Original Purpose: The focus shifts from understanding complexity to managing time

Whats the solution?

In the fast-paced world of software development, traditional management practices often fall short. The need for flexibility, innovation, and efficiency has led to the rise of new methodologies that prioritize team autonomy and streamlined processes. Two such concepts are empowering engineers with time control and adopting the #NoEstimates approach. Let’s explore how these ideas can transform your development process.

Giving Time Control to Engineers

Traditional management often involves strict time tracking and deadline enforcement. However, this can lead to stress, reduced quality, and a lack of ownership among engineers. Instead, consider giving engineers control over their time, much like Toyota did with its lean management practices.

The Toyota Example

Toyota revolutionized manufacturing by handing the stopwatch to its employees, allowing them to manage their own time. This shift created an environment where workers could focus on quality and efficiency, leading to better outcomes.

Benefits for Software Teams

• Increased Ownership and Motivation: Engineers feel more responsible for their work, leading to higher engagement and satisfaction.
• Improved Quality: Without the pressure of arbitrary deadlines, engineers can focus on delivering high-quality code.
• Natural Work Rhythm: Teams can work at a pace that suits them, leading to more sustainable productivity.
• Enhanced Problem-Solving: Engineers have the time to address root causes rather than just symptoms.

Planning Without Estimations: Projections

Estimating stories in terms of hours or points has long been a staple of Agile planning. However, this process can be time-consuming and often inaccurate. Enter the #NoEstimates approach, which focuses on counting stories instead of estimating them.

How It Works

Instead of estimating the time required for each story, simply count the number of stories and track how many are completed each week. Use this data to project release dates:

Projected End Date = (Total Stories / Stories Completed per Week) + Start Date

Why It Works

• Reduces Planning Overhead: Less time spent on estimation means more time for actual development.
• Increased Accuracy: Counting stories can be as accurate, if not more so, than traditional estimation methods.
• Focus on Value Delivery: Teams concentrate on delivering valuable features rather than meeting arbitrary time estimates.
• Eliminates Anchoring Bias: Without estimates, teams are less likely to be influenced by initial predictions.

The Science Behind It

The #NoEstimates approach leverages the law of large numbers, where variations average out over many stories. This method removes cognitive biases from the estimation process and often results in actual throughput that exceeds theoretical estimates. By focusing on flow rather than deadlines, teams can deliver more consistent and reliable results.

Conclusion

Empowering engineers with time control and adopting the #NoEstimates approach can lead to a more motivated, efficient, and effective development team. By shifting the focus from time management to value delivery, organizations can create an environment where innovation thrives.

Speculation: As software development becomes increasingly modular, with smaller and more frequent story deliveries, these approaches could become even more effective.

Resources:

• https://ronjeffries.com/articles/019-01ff/story-points/Index.html
• Vasco Duarte https://softwaredevelopmenttoday.com/#