How to calculate vaccine effectiveness

How can it be that more vaccinated people than unvaccinated people die of a given disease? It sounds paradoxical, and it even sounds a bit as if vaccination doesn’t work — or even does harm — but we’ll see that this need not be true.

Imagine a town with a hundred thousand inhabitants, hit by a vicious and treacherous disease called Provid. Scientists and researchers have managed to develop a vaccine that protects against the disease, and part of the population has been vaccinated with it. Yet when we look at the data, 1,000 vaccinated people and 1,000 unvaccinated people have died of Provid. The townspeople are confused and feel that the vaccine doesn’t work, since as many vaccinated people die as unvaccinated. So does the vaccine work or not?

Hard to say 🤷 from just the data we have so far. It isn’t enough to know how many vaccinated and unvaccinated people died of Provid. We also need to know how many people actually got vaccinated. I could ask you: which is more dangerous, driving a car or climbing Mount Everest? The death figures speak clearly: hundreds of people have died trying to climb Everest. In cars, over a million people are killed. Every year. So it’s obviously safer to try to climb Everest than to get into a rolling coffin.

A warning light should be going on in all your heads now — in reality, climbing Everest is not safer than driving. The numbers mislead us because we haven’t taken into account that while thousands of climbers have gone up Everest, billions of people travel by car. In the same way, we have to take into account how many people in our imaginary town are vaccinated. Let’s work through a few examples:

• What if we had vaccinated only 1,000 people in the town, and 99,000 were unvaccinated? That would mean that every vaccinated person died, while only 1,000 out of 99,000 unvaccinated people died. The vaccine would not only not help, it would do harm!
• What if it were the other way round, with 99,000 vaccinated people in the town and only 1,000 unvaccinated? Every unvaccinated person would have died, while among the vaccinated only 1,000 out of 99,000 would have died. The vaccine may not be 100% effective, but it clearly improves the chance of survival by an order of magnitude.
• What if half of the town were vaccinated and the other half weren’t? Then 1,000 out of 50,000 vaccinated people died and 1,000 out of 50,000 unvaccinated people died. In both groups 2% of the people died — which means the vaccine has no effect, either positive or negative.

What is vaccine effectiveness

So how do we calculate the effectiveness of a vaccine? We say that vaccine effectiveness is the proportion of people whom the vaccine saved from death. For example, suppose 1,000 unvaccinated people died of the disease, while in an alternative reality where we vaccinated those same 1,000 people, only 300 died and 700 survived. In that case the vaccine saved 700 people out of 1,000, so its effectiveness would be 70%. If nobody died, the effectiveness would be 100%; if everybody died, the effectiveness would be 0%.

Fine, but we don’t have two parallel universes in which we vaccinate someone in one and don’t vaccinate the same person in the other. So we have to calculate effectiveness differently. Let’s go back to our town of a hundred thousand inhabitants. Let’s add the missing piece of information — say we vaccinated 90,000 people. This is data we can collect in the real world too. What is the effectiveness of the vaccine? First we work out the proportion of deaths in each group, that is, what percentage of vaccinated and unvaccinated people died:

• 1,000 out of 10,000 unvaccinated people died, which is 10%. Let’s denote this by NL = 10% (not vaccinated).
• 1,000 out of 90,000 vaccinated people died, which is roughly 1.11%. Let’s denote this by OL = 1.11% (vaccinated).

We see that 1.11% of the vaccinated people died, while 10% of the unvaccinated people did. We calculate the resulting effectiveness U using the formula

$$U=\frac{NL-OL}{NL}\cdot 100\mathrm{\%}$$

Substituting, we get

$$U=\frac{10-1.11}{10}\cdot 100\mathrm{\%}=\frac{8.89}{10}\cdot 100\mathrm{\%}=0.889\cdot 100\mathrm{\%}=88.9\mathrm{\%}.$$

The effectiveness of our vaccine is 88.9%. In other words, the vaccine saved 88.9% of the people who would otherwise have died. We can also check this: in the unvaccinated group 10% of people died. If we hadn’t vaccinated those 90,000 people either, the same 10% of them would have died, which is 9,000 people. Since the vaccine is 88.9% effective, 88.9% of those 9,000 people ended up surviving thanks to the vaccine, that is, 88.9% of 9,000 is exactly 8,000, and that leaves 11.1% of people who sadly died anyway — the 1,000 deaths from the question.

We see that this vaccine gives us a much higher chance of survival, but it can’t be said that after vaccination we are 100% certain not to die of the disease.

The more people vaccinated, the more vaccinated people die

Incidentally, this leads to an interesting paradox. Let’s imagine that we vaccinate another 5,000 people in the town with our 88.9% effective vaccine, so that we reach 95,000 vaccinated people and only 5,000 unvaccinated. Then

• Of the 5,000 unvaccinated people, 10% die, that is, 500.
• Of the 95,000 vaccinated people, 1.11% die, that is, roughly 1,055.

‼️‼️‼️ SHOCK ‼️‼️‼️ Many more vaccinated people die than unvaccinated. But that is purely because there are far more vaccinated people than unvaccinated. Another way to put it: while there are 19 times as many vaccinated people in the town as unvaccinated, we have only twice as many deaths among the vaccinated as among the unvaccinated.

And above all — the total number of deaths has fallen. While now we have 500 + 1055 = 1555 deaths, in the previous example we had 1000 + 1000 = 2000 deaths.

Different values of vaccine effectiveness

Let’s look at some interesting values of effectiveness:

• We would get 100% effectiveness if OL were equal to zero (none of the vaccinated people died) while the unvaccinated people did die. The effectiveness of a vaccine can’t be greater than 100%.
• We would get 0% effectiveness if NL = OL (if the same percentage of vaccinated and unvaccinated people died). It would mean that the vaccine has no effect.
• We would get negative effectiveness if OL were greater than NL, that is, if a higher percentage of vaccinated people died than unvaccinated. In that case the vaccine would do harm.

You can picture a vaccine that is around 83% effective like this: suppose you fall ill and are unlucky enough that the disease would have finished you off. The vaccine still gives you extra protection — you can roll a die, and if you roll a one to a five, the vaccine saves you, but if you roll a six, the disease kills you anyway.

Beware of different groups of people

Be careful: when calculating vaccine effectiveness, it isn’t a good idea to follow the formula I gave above blindly. To apply it, it helps to know how the disease behaves and what our vaccination strategy is. Diseases differ, they attack different groups of people, and likewise different groups of people will have different vaccination coverages. If a disease attacks only women, we may not care what the vaccination coverage among men is. If a disease attacks older age groups, we may not care what the vaccination coverage among younger age groups is, and it will be better to calculate the effectiveness using the vaccination coverage among the older age groups.

Likewise, we can measure different types of protection for vaccines. Throughout this article we have, rather morbidly, talked only about death, but for vaccines we can also measure how much a vaccine reduces (or increases…) the chance that we fall ill, that we pass the disease on, or, say, that we need hospital treatment.