Calibration is a practice, not a talent

score yourself, or you are just vibing

Good forecasters are not born knowing what seventy percent feels like. They learn it the slow way, by writing down predictions and grading them.

2 min read
  • calibration
  • scoring
ALL NOTES

There is a comforting myth that some people just have a feel for probability, the way some people have perfect pitch. It lets the rest of us off the hook. If calibration is a talent, then not having it is not our fault.

It is not a talent. Calibration is the plain, measurable skill of having your confidence match reality: when you say seventy percent, the thing happens about seventy percent of the time. Nobody has that by default. The people who have it built it the same way anyone builds a skill, by doing the thing, checking whether it worked, and adjusting. The checking is the whole game, and it is the part almost everyone skips.

You cannot improve a number you never wrote down

The reason most people never get calibrated is simple: they never find out how they did. A forecast made in your head and never recorded cannot be graded, and an ungraded forecast teaches you nothing. Worse, memory quietly rewrites it. The call that went wrong becomes “well, I always had doubts,” and the lesson evaporates.

So the first move is boring and non-negotiable. Write the prediction down, with a number and a date, before you know the answer. That record is the only thing standing between you and a comfortable story about how you basically saw it coming.

Ranges beat points

A single number pretends to a precision you do not have. “The project ships March 14th” will be wrong, and being wrong will feel like failure even when your thinking was sound. A range is honest about what you actually know.

  • p10 is your optimistic-but-plausible edge: only a one-in-ten chance the answer comes in below it.
  • p50 is your genuine middle: as likely to be too high as too low.
  • p90 is the pessimistic edge, with a one-in-ten chance of coming in above.

Forecasting the spread instead of the point does two things. It makes you say out loud how uncertain you are, and it gives you something fair to grade later. A point estimate is either right or wrong. A range you can actually score.

Being uncertain and saying so is not weakness. It is the entire skill. The goal is not to be sure, it is to be right about how sure you are.

Keep score without flinching

The uncomfortable part is looking back. Pull up the predictions that have resolved and sort them by confidence. Of everything you called eighty percent, did about eighty percent happen? If far more did, you are underconfident and can push harder. If far fewer did, you are overconfident, which is the more common and more dangerous direction.

Do this a few times and something shifts. Seventy percent stops being a vibe and starts being a quantity you have a feel for, because you have watched a hundred of your own seventy-percents play out. That feel is what people mistake for talent. It is just practice with the scoreboard left on.

Try it

Put your open questions on a board as trackers, each with a p10, p50, and p90. Come back when they resolve and mark where the answer actually landed. The gap between your bands and reality is your calibration, drawn to scale. Start tracking.