COVID Vaccine Efficacy – uncertainty reigns

Do you know the vaccine efficacy numbers cited for COVID vaccines, the ones coming from the pre-release trials…? I’ve seen numbers in the 65-95% ball park, as e.g. in this Lancet article which is IMO better understood by reading this blog post.

A few days ago I noticed the below data from Israel, on Covid cases among vaccinated vs non-vaccinated:

Simply by eye-balling the top table, we see that most COVID cases now occur among the vaccinated. By a huge margin. For each age group.

That doesn’t look good…! But there are some caveats: first, look at the column “Percent of Population Vaccinated” : the vast majority of people in Israel are now vaccinated, so unless we expect the vaccines to be 100 % efficient (which I in fact myself believed vaccines being, before COVID vaccines came into picture…) in blocking the virus, chances are that most of the infected will indeed come from the larger cohort (those vaccinated). Secondly, the table does not provide the group sizes, so we cant calculate the incidence rates for the vaccinated vs non-vaccinated. Thirdly, the absolute number of cases (relative to the unknown group sizes) is very small, so the uncertainty with this few data points on cases, particularly for the non-vaccinated, is huge.

But we can still do a back-of-the-envelope type of calculation on vaccine efficacy – as long as we keep in mind the uncertainty coming from the small numbers – by assuming the group population size, and I’m going to be lazy and set it to 1M per age group.

With a bit of arithmetic, we get to:

incidence & efficacy based on assumed population size

Efficacy is given by the rightmost column. It’s way below the 65-95% range given e.g. by the Lancet article mentioned above. However, let’s not forget that this was calculated with very little data, so we should be very careful drawing any conclusions from this data, until we have more data on cases.

One way to see how certain / uncertain these numbers are can be obtained by running a Bayesian analysis, to obtain not only point estimates (as above), but full probability distributions for the efficacy rates for the various age groups.

I did a quick & dirty version of such an analysis for 7 of the 8 age groups above (the 90+ group gets an infinite negative efficacy since there are 0 cases in that group within the non-vaccinated):

One way to understand the uncertainty is to look at the 89% Credible Interval, given by the black horizontal bar: it crosses zero for all age groups, meaning that there’s some probability (density) on both sides of vaccine efficacy being positive or negative. And looking at the last graph, the one for 80-89 year olds, where we have only 23 cases for vaccinated, and 2 for non-vaccinated, the credible interval is almost perfectly balanced around zero, meaning that we should probably not put much trust in the efficacy number for that age group.

Nevertheless, it seems that vaccine efficacy in Israel now, when most Israelis are vaccinated, does not reach even close to the 65-95% range given by the pre-release testing. Far from it.

UPDATE: here’s similar findings based on UK data

About swdevperestroika

High tech industry veteran, avid hacker reluctantly transformed to mgmt consultant.
This entry was posted in Data Analytics. Bookmark the permalink.

2 Responses to COVID Vaccine Efficacy – uncertainty reigns

  1. Brad and Rachel Forwell says:

    This is interesting, Tommy. What a mess we have created for ourselves. If these vaccines don’t provide the magic bullet that many expected/hoped, and if unrealistic expectations towards public health continue (e.g. the false belief that we can stop an endemic virus from circulating), then we are all in for a very long and unnecessary ride.

    • Yes. Snap decisions, eg trying to min/max a single parameter, without considering all the dependncies and feedback loops, whenever dealing with complex systems is a recipe for disaster.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s