Research/Software
Agile forecasting's 85% delivery dates came true 80% of the time
In a pre-registered test on 25 public Apache Jira projects, the Monte Carlo forecasts behind many agile tools were too optimistic. Correcting each project's odds from its own track record brought its 85% statements back to about 85%, but did not make every probability more accurate.
Nov 19, 2026 · 6 min read

Many agile teams now promise delivery dates with a Monte Carlo forecast. They resample how many work items they finished on past days, simulate thousands of futures, and quote the date they will hit "with 85% confidence". We tested whether those statements come true as often as they claim, on two years of real history from 25 public Apache Software Foundation projects. They did not: the 85% statements came true 80% of the time at a two-week horizon and 75% at four weeks. 23 of the 25 projects fell short. We then tested a fix built into MV2's open-source Jira app Ship Odds: recalibrating each project's odds from its own record of past misses. That brought the 85% statements back to 84.7%. It did not, however, make the forecasts better probabilities across the board, and we report that too.
The question
The "85% date" is sold by several Jira apps and recommended in practitioner books on flow metrics. A forecast that says 85% should come true about 85 times in 100. If it comes true 75 times, the team is promising its customers more certainty than it has. We found practitioner write-ups, one company blog post with calibration heatmaps for three anonymized teams, and academic work on predicting Apache iteration delivery. As far as we found, there was no public, pre-registered, multi-project test of this exact procedure with open code and data, and no test of a tool that corrects itself from its own track record.
What we did
Both studies were pre-registered on 17 November 2026, before any data was fetched, with eleven predictions fixed in advance. We counted issues resolved as Fixed per day in the public Apache Jira from November 2024 to November 2026. Of 50 candidate projects, the 25 with at least 200 such issues were kept, including Kafka, Spark, Flink, Cassandra and Solr. Every 7 days the model saw only the past 90 days and stated, at 50%, 70%, 85% and 95%, how many items would be finished at least in the next 14 or 28 days. We then checked each statement against what really happened. Projects were weighted equally.
The correction is standard probability-integral-transform recalibration, shrunk toward no change so that a short history corrects only a little. It was learned strictly out of sample: at each point it used only earlier forecasts of the same project whose outcomes were already known.
25
public Apache projects, two years of history
80.0%
of 85% statements came true (14 days)
84.7%
after self-correction (raw 79.3% on the same forecasts)
8 of 25
projects with a better Brier score after correction
What we found
The raw forecasts were too optimistic at every setting we tried. The miss grew with the horizon: 75.1% at 28 days. A version that resamples whole weeks instead of single days did a little better (82.7%), as predicted. Contrary to our prediction, a shorter 30-day memory was slightly better than 90 days, not worse. The 95% statements came true 90.5% of the time, just missing our predicted cut-off of below 90%. Four of six part-1 predictions held. The headline prediction (below 80%) held by 0.0002, so read it as "about 80%".
How often "85%" statements came true
Part of the gap is built into the method. On synthetic teams whose daily output really is drawn from a fixed distribution, which is exactly what the model assumes, the same statements came true 84.9% of the time against a stated 87.8%. Treating 90 noisy days as the true rate costs about 3 points. On real projects the shortfall is about 8 points, so roughly 5 points come from teams' pace changing over time.
The self-correction did what it was designed to do for the dates teams promise. Four of five part-2 predictions held.
85% statements that came true, raw versus self-corrected
The failed prediction matters most for anyone building such a tool. We predicted that at least 60% of projects would get a better Brier score, which scores every probability rather than just the 85% line. Only 8 of 25 did. The average still improved slightly (0.1822 to 0.1808), but almost all of that came from a few projects where the raw model was badly wrong. On most other projects the correction added a little noise. Synthetic controls showed the same pattern: no benefit for stable teams and a clear gain for teams whose pace drifts down.
Brier score per project at 14 days (lower is better): largest gains and losses
| Project | Raw | Corrected | Raw 85% hit rate |
|---|---|---|---|
| HUDI | 0.343 | 0.255 | 47% |
| AMQ | 0.178 | 0.153 | 78% |
| ORC | 0.239 | 0.221 | 68% |
| KAFKA | 0.223 | 0.218 | 71% |
| THRIFT | 0.139 | 0.147 | 86% |
| HIVE | 0.163 | 0.173 | 82% |
| SOLR | 0.170 | 0.182 | 82% |
What it means
If your tool says 85%, plan for something closer to 80% at two weeks and 75% at four weeks, unless you have checked your own record. Learning from that record fixes the promise: the corrected 85% line held 84.7% of the time at two weeks and 81.2% at four. The cost is honesty in numbers, with corrected 85% commitments about 14% smaller than raw ones. A correction applied blindly to every team is not a free improvement. The next step, which we will pre-register before testing on projects and trackers not used here, is to correct only when a team's own record shows a clear miss. Ship Odds, MV2's entry in Atlassian's Codegeist 2026 hackathon, keeps the corrected 85% and 95% dates and shows each team its own track record so it can check this for itself.


