Estimating Work

I recently put together a prioritized list of technical to-dos for a client. On one of them I put a note of “This will be a heavy lift”. The CTO appropriately asked “Can you unpack that a bit more for me?”

This will be a heavy lift.

Estimation is Hard

The exchange got me thinking about estimating work which has long plagued the software industry. In my 20 years of development experience the process of estimating work has largely seemed to involve a crystal ball, wishful thinking, and downright lies. Often a combination of more than one of those. Oh I know your Agile Champion in the company will say it’s “not that hard” or begin a retort with something like “Well, actually in Agile….” and then finish that statement with words that you and I know to be complete nonsense detached from reality but fits a narrative he or she is paid handsomely to continue pushing.

That doesn’t mean we can’t have reasonable conversations about the size and scope of work. Even as an experienced engineer I can’t tell you exactly how long something is going to take beyond extremely trivial tasks. But I can tell you with great confidence what the scope of a task or project will be and whether it will take hours, days, weeks, months or years. Some refer to this as t-shirt sizing although I feel like that doesn’t adequately capture the nuance involved. Anyone can guess between 4 sizes and be right 25% of the time. 😆

At a previous company, I was involved in discussions about a potential re-write of an entire portion of the backend. The inexperienced team was convinced it would be done in 6 months. “You’re talking about 3+ years minimum.” That was my input that was laughed at and immediately ignored. After 5 years with that company the rewrite still hadn’t been delivered. Not a line of new code was running in production.

A Conversation Starter

So what do I mean when I say “That will be a heavy lift.”? This is a reasonable question and made me think of the following illustration.

quadrantChart
    title How Heavy a Lift?
    x-axis Low Complexity --> High Complexity
    y-axis Small Size --> Large Size
    quadrant-1 Months to Years
    quadrant-2 Weeks to Months
    quadrant-3 Hours to Days
    quadrant-4 Days to Weeks
    String Change: [0.1, 0.1]
    Rename a Variable Everywhere: [0.15, 0.4]
    New Product: [0.81, 0.78]
    The Rewrite: [0.85, 0.9]
    Switch DB Providers: [0.45, 0.7]
    "Increase Test Coverage" : [0.17, 0.8]
    Refactor a component: [0.53, 0.4]
    Fix a race condition: [0.67, 0.1]
    Fix a memory leak: [0.58, 0.18]
    Port to a new MCU: [0.59, 0.57]

Unit Tests - Taking your code base from 5% coverage to 80% isn’t particularly complex (although it can be for sure) but it is a LOT of work. You may have to write thousands of tests. It’s large in size but low in complexity.

Fix a Race Condition - This often boils down to a few lines of code. The PRs are often small but the amount of time spent finding the race condition source can take weeks.

Memory Leaks - Similar to race conditions. Most of the time is spent finding the problem. Normally a few lines of code fixes it. It’s small in size, but complex work.

The Rewrite - You have grand plans to solve all your technical debt woes. The problem is, the scope is the entirety of your current business value. It’s a huge amount of complex work. You better be in it for the long haul.

Now before you disagree with the placement of EVERY point on this chart remember this is just for illustration purposes and will depend on things like company culture, state of the codebase, size of the team, etc. It’s meant to foster a conversation about the relationship between amount of work vs complexity of work and how they affect time of work. All factors that Agile will tell you are “factored in” to their estimation techniques. Reality tells a different story.

So What’s a Heavy Lift?

A “heavy lift” doesn’t necessarily imply a task is complex or difficult. It just means it will take a lot of time. That could be a rewrite or it could be renaming a common variable everywhere in a 5 million line codebase.

When I worked at Microsoft a grizzled veteran often said “Nothing can be done in less than a day.” This seems like an exageration but think about how your company works. The change could be a one-liner. But then you have to write unit tests. You have to document it. You need to stage it and notify QA to test it. You have to take it to the change advisory board for approval and then you have to deploy it. This may sound extreme but I worked at a company where a string change took the entire day to “complete”. I’ve also made string changes in my one-person company that took 5 minutes.

I mention that because the graph above is highly dependent on your company and work environment. How much bueracracy is involved? Are there political minefields you have to navigate? Maybe you just have that one co-worker that likes to take 3 hours to code review a 10 minute change. All of these things affect what “heavy lift” means for you and where the dots would be on your company’s graph.

The last point I want to make is that the further away from the bottom left of this graph you get, the more inaccurate your predictions will be. It is hard to predict, with great accuracy, how long it will take to deliver a large, complex project.

Conclusion

When estimating work consider the difference and relationship between a lot of work and difficult work. They both heavily influence how long something will take. Being able to articulate the nuances involved in your company will help immensely when communicating with stakeholders which, like it or not, is part of your job as an engineer.