6 Hidden Costs of Large Waterfall Projects
When you come up with a good idea you want to release it to customers as soon as possible. For years marketing, customer services and product teams have been pushing for quicker and quicker turnarounds.
And IT departments want to increase speed as well but they are often hampered by the way of working.
I was speaking to an organisation’s Head of Delivery recently who said that he wants to move to agile releases. He has been pushing his teams to split big projects into smaller releases so that value can be delivered quickly but the cost of delivery started increasing and according to him “became prohibitive”.
The problem is that the cost is only being measured in one way – the cost of delivery of a single project. But nothing happens in isolation, so to measure the real cost you need to factor in the costs across the portfolio. Which brings me to the 6 hidden costs of large projects.
1. Building the Wrong Thing
We’ll start with the biggest cost. Yes, it might cost a bit more to release more frequently, but how much could you save by learning that you are on the wrong track and shifting your focus before you build the wrong thing?
Up to 66% of features fail to deliver the expected value so, even without any of the further hidden costs, this alone is enough to warrant the extra investment.
2. Holding Cost
Holding cost refers to the revenue/value that the business is losing by not releasing product features or fixes that are ready to be released.
When a project is approved it typically has a return that is a multiple of the investment – the business value far outweighs the implementation cost. So even if you have to increase the cost of delivery slightly the early release will likely mean that the business is better off.
3. Merge Costs
Projects don’t happen in isolation. In the company I am currently in, for example, there are over 60 projects running concurrently. This means that there is a constant need to merge code changes from projects that are delivering now into long-running projects.
Each merge carries risk but bigger merges have an exponentially larger risk which results in merge costs increasing exponentially with project size and duration.
4. Delaying Costs
Another issue that can occur is premature dependency enforcement. Let’s say you want to start a new project and you think it will take 6 months to deliver.
There is already another project in flight that will go live in 5 months time so it makes sense to build your changes on a branch of the earlier projects code. This reduces the merged cost above but if the earlier project gets delayed then you’re delayed too.
5. Blocking Costs
With monolithic architectures, which are still the norm, there is only one configuration of code and infrastructure in production so only one project can be on the path to production at a given time.
This means that other projects must queue up behind it. Larger projects have longer paths to production durations which block other projects from releasing.
6. Planning Costs
When organisations run a lot of projects, they are often scheduled very close to one another to take advantage of the limited release windows across the portfolio.
This causes any project slippage to have a cascading effect on subsequent projects. I refer to the resulting chaos as circular planning; align all of the variables to generate a plan that works, start again when one dependency slips and repeat continuously until you finally release.
Once you factor in the number of projects in the portfolio and the planning, resource management and communication overhead the cost of a single project slippage multiplies quickly.
7-(ish). Change Requests
Change Requests are obvious, but contentious cost. The larger the project the more change requests will be required along the way. This is so accepted that many companies include change request contingency into their project planning.
The problem with this cost is that it isn’t just the change request – it is the analysis time, the approval time and the rework time. While this is significantly more expensive in Waterfall projects I’ve left it out because agile projects do encounter change as well.
How Can We Reduce These Costs?
Most costs are related to the size of the project. Smaller projects reduce these costs because features are released as soon as they are developed, merges become smaller, the risk of delays reduces and the pipeline remains open.
The counter-argument is that the cost of regression testing increases because you have to make sure that you test everything on every small release.
Given the numerous benefits of small releases, the focus needs to shift to doing fewer projects (and therefore less regression testing) to reducing the cost of regression testing.
The challenge is that the processes that are ingrained in many organisations have a bias towards large projects, as is the case in the company that I mentioned at the beginning. If we want to enforce smaller projects we need to introduce “forcing functions” that break old habits. There are two types of forcing functions that we can use:
Imagine if you could only spend one week on your path to production. What impact would that have on your projects? To complete your regression testing you need to invest in automation or perform risk-based testing. Once these improvements are delivered and everyone is adhering to the one week limit, you can reduce the time limit to 2 days, triggering further innovation.
Basically, we need to make the costs of the way we are working visible or they will continue to be ignored. Taxes are a great way to achieve this goal. A tax that increases exponentially in line with the length of a project brings the cost of delivery more inline with their true costs. The outcome will be that people will start asking for smaller projects, or in the cases where you have to do a large project, the “tax” will fund the necessary technology and ways-of-working improvements so that future projects won’t be as expensive. Either outcome enables smaller projects.
The Curse Of Getting What You Ask For
Making projects smaller and easier to release enables the industry-wide shift towards product teams, which essentially remove the concept of projects altogether.
Teams are continuously releasing small iterations and making constant improvements to the systems they look after. This is a much better structure for business outcomes because you can test ideas quickly and pivot where necessary.
The downside is that this is much more involved for the marketers or other business representatives. You can’t just handsome ideas over to someone else to implement.
You need to share your objectives with the product teams and work with them, continuously, as you try out new experiments to see what works and what doesn’t. While there is a lot more investment on your behalf it is really invigorating to see ideas turn to reality quickly.
If you’re interested to learn more, check out UXDX which is focused on helping companies shift from traditional waterfall projects to cross-functional product teams.