Striving for Imperfection

A look into what it means to be a "leaky" vaccine.

Correction: A correction was made to the definition of a leaky vaccine with “degree of effectiveness” removed and replaced with “infection reduction”.

Let’s propose a situation relating to vaccine clinical trials.

Say that a vaccine is reported to have a clinical efficacy of 95%: does that mean that it reduces disease and severity of illness by 95%, or does it mean that out of 95% of all interactions with other people you’ll be perfectly safe while you may get sick from the other 5% of interactions?

Numbers may sometimes be easy to work with, but other times they may boil down highly complex issues into a manner that becomes more reductive than productive. It also doesn’t help when other terms such as “leaky” vaccine are thrown in as well.

We certainly know that these COVID vaccines are “leaky”, and that it certainly is a problem. So what does all of this mean, and why is this such an issue?

A Brief Breakdown of Vaccines

The flavors of vaccines

Most vaccines are described by the type of pathogen they target, yet it’s only recently that the layperson has begun to describe vaccines based on their level of effectiveness.

There are variations of vaccine descriptions in the literature, and most of them are derived from mathematical modeling and variables used to map out a vaccine’s effectiveness. For our sake, we’ll exclude all of the math and focus on the descriptive aspect.

Generally, there are 3 types of vaccines which are classified by their immune failures:

• “Leaky” vaccines: Leaky vaccines are an example of a vaccine in which the effectiveness is based on degree of infection reduction. These vaccines are not sterilizing, but may infection rates for those that receive them. These vaccines also operate on a per-exposure basis, and their effectiveness is heavily contingent upon this factor, as well as many individual recipient factors. The annual influenza vaccine is an example of a leaky vaccine, as well as the current COVID vaccines and malaria vaccines.

  • Note: The definition above ran into a circular argument where I placed the phrase “degree of effectiveness degree” or something similar to that. I changed the terminology to infection reduction as bolded above. Apologies for the strange wording and awkward definition.

• Sterilizing vaccines: Sterilizing vaccines are also called “all-or-nothing” vaccines because their effectiveness is wholly dependent upon the vaccine’s ability to “take” to the recipient- either the vaccine elicits an immune response (in which case you are fully protected) or the vaccine does not take and you are provided no protection. For example, a sterilizing vaccine may provide protection to 80% of the recipients, which means that 80% are fully protected while the other 20% are left vulnerable. Unlike leaky vaccines, sterilizing vaccines are not dependent upon exposure, although a few of the factors affecting leaky vaccines may affect sterilizing vaccines as well. Measles vaccines tend to be a typical example of a sterilizing vaccine.

• Waning vaccines: The third category is waning vaccines, in which the vaccine may provide a high degree of protection close to the point of inoculation but begins to lose effectiveness weeks and months out. Cholera vaccines are an example of waning vaccines, and to a good degree so too are the COVID vaccines. Waning immunity will not be discussed in this article.

Factors influencing vaccine effectiveness

On an individual basis, several factors are continuously at play that alter a vaccine’s effectiveness. Most important is whether a vaccine is leaky or an all-or-nothing (AoN) vaccine, which both exist on opposite ends of the vaccine efficacy spectrum.

For the most part, AoN vaccines are the most effective for those in which the vaccine takes due to the fact that AoN vaccines act independent of exposure risk- it doesn’t matter how many infected people you come across- you should still be protected under all circumstances (as long as the vaccine takes).

But if we were to take exposure into account, we can see that leaky vaccines greatly lose effectiveness when the risk of exposure increases (Edlefsen, P. T.¹).

It has been shown that the mechanism of the vaccine's protection is relevant to the relationship among these kinds of efficacy, with “leaky” vaccines (defined as those modifying per-exposure infection rates for all subjects equally) at one extreme and “all-or-none” vaccines (which completely protect some subjects and have no effect on the others) at the other extreme. While for all-or-none vaccines the overall attack rate is reduced by the fraction of recipients that have protective responses, for leaky vaccines the attack rate is reduced by an amount that depends on the number of exposures that each subject experiences.

Without using numbers, let’s analogize this with the commonly used scenario of a car accident and wearing seatbelts. Suppose that wearing a seatbelt in a car accident reduces the risk of injury or death. That’s great when the incident only happens once. But let’s say that it’s not a rare occurrence and you just happen to be that person who just does not know how to drive well, no matter how enthusiastic and endearing your driving may be. Instead, you get into 10 accidents over the course of a year.

Aside from looking at your monthly insurance premiums, we can argue that each successive car accident carries with it a risk of dying or becoming severely injured. You may be lucky after the first accident, but how lucky would you be over the course of 10 accidents? Each accident carries with it its own risk, and each additional accident increases the risk of injury or death.

The same rules apply to vaccines dependent upon exposure (i.e. leaky vaccines). In one incident you may have a high degree of protection, but each exposure to an infected person carries with it a risk of getting sick. Multiply that over multiple exposures and the effectiveness of stopping infection greatly wanes.

Essentially, as the risk of exposure increases, whether due to individual behaviors, high levels of circulating virus, or high transmission rates, the effectiveness of a leaky vaccine greatly declines.

Additional factors to take into account can be seen in the diagram below (Ragonnet, et. al.²):

From Ragonnet, et. al. Direct measures indicate the effectiveness of a vaccine on the individual while overall effectiveness, which takes into account both direct and indirect effects of vaccination, indicates overall effects on public health. Most clinical trials evaluate a vaccine’s effectiveness based on direct effects rather than overall effects, yet overall effects are what is used to drive public health measures. R0= reproductive number (how many people can one infected person go on to infect), Relative infectiousness (c) refers to how likely someone who is infected is to go on and infect another person.

There are many caveats to this modelling³, but the gist is that overall vaccine effectiveness generally trends with higher vaccine coverage, low reproductive number of the pathogen in question, low risk of re-infection, and low infectiousness of an infected person. All work together to provide the overall perception of what we consider to be a vaccine’s effectiveness, and under ideal circumstances leaky vaccines are no different that AoN vaccines.

But once again, when infection rates are high and even vaccinated individuals can transmit the virus, then leaky vaccines lose most of their effectiveness (Ragonnet, et. al, emphasis mine):

We demonstrate that ‘AoN’ vaccines are consistently at least as effective as ‘leaky’ vaccines, but prove to be much more effective under conditions where diseases are more difficult to eradicate. In particular, ‘AoN’ vaccines are most beneficial when risk of re-infection is high, when the infection is highly transmissible or when the vaccine has low efficacy. Even though it is difficult to determine which mechanism is associated with a given vaccine in practice, this study provides insights into why population-level vaccination failure may occur despite significant biological efficacy. ‘Leaky’ vaccines in particular may have an overall effectiveness that is significantly lower than the vaccine efficacy in certain configurations. Furthermore we demonstrate that the impact of such assumptions regarding vaccine mechanism are not trivial, suggesting that it may be worthwhile considering different scenarios when modelling vaccination against infectious diseases.

So what does this mean in the context of COVID vaccines? For one, we know that these vaccines are leaky and not sterilizing, meaning that the overall vaccine’s effectiveness is contingent upon all the variables outlined above to a greater extent than AoN vaccines.

Because the virus is still widely circulating, that Omicron base reproductive value is high, and that transmission is not halted by vaccination, we can make a clear argument that any effectiveness of the vaccines would be greatly reduced even without examining the neutralizing capabilities of antibodies. Considering that Omicron breaks all of the rules (albeit abstract rules) required to have an idealized leaky vaccine we shouldn’t be surprised that we are witnessing the vaccine fallout happening today. Leaky vaccines are only as good as the virus it targets, and it’s quite obvious that these aren’t the vaccines worth implementing to target SARS-COV2.

But then how can we explain the supposedly “phenomenal” clinical trial results? Are the vaccines just beginning to lose effectiveness in the wake of COVID, or were there always problems with the trials?

Faulty Clinical Trials; Faulty Vaccines

Talking about vaccine clinical trials is no different than beating a dead horse at this point. We know the trials had very few participants and suffered from many methodological issues.

So instead this section will contextualize a bit of what we described above and see where else these trials went wrong.

One thing that we know has been a lie (after we found out that these vaccines were not sterilizing) was that these vaccines would reduce severe illness and death. Yet that was never the case for the clinical trials- they were never designed to measure hospitalizations and death. Instead, they were always set up to only measure minor illnesses.

Taking us back to 2020 Peter Doshi raised these concerns in an October 2020 bmj article⁴:

In a September interview Medscape editor in chief Eric Topol pondered what counts as a recorded “event” in the vaccine trials. “We’re not talking about just a PCR [polymerase chain reaction test]-positive mild infection. It has to be moderate to severe illness to qualify as an event, correct?” he asked.8

“That’s right,” concurred his guest, Paul Offit, a vaccinologist who sits on the FDA advisory committee that may ultimately recommend the vaccines for licence or emergency use authorisation.

But that’s not right. In all the ongoing phase III trials for which details have been released, laboratory confirmed infections even with only mild symptoms qualify as meeting the primary endpoint definition.9101112 In Pfizer and Moderna’s trials, for example, people with only a cough and positive laboratory test would bring those trials one event closer to their completion. (If AstraZeneca’s ongoing UK trial is designed similarly to its “paused” US trial for which the company has released details, a cough and fever with positive PCR test would suffice.)

Part of the reason may be numbers. Severe illness requiring hospital admission, which happens in only a small fraction of symptomatic covid-19 cases, would be unlikely to occur in significant numbers in trials. Data published by the US Centers for Disease Control and Prevention in late April reported a symptomatic case hospitalisation ratio of 3.4% overall, varying from 1.7% in 0-49 year olds and 4.5% in 50-64 year olds to 7.4% in those 65 and over.13 Because most people with symptomatic covid-19 experience only mild symptoms,14 even trials involving 30 000 or more patients would turn up relatively few cases of severe disease.

And so these studies were never able to properly measure severe illness through hospitalizations and death. Instead, most measures of infection were done through routine rt-PCR testing or reports of symptomatic infection. We may make an argument that lack of mild infections would infer lack of hospitalizations and death, but that inference relies on the notion that these two subsets are similar demographics-wise.

Instead, we may be more inclined to argue that younger and healthier participants would be likely to suffer mild disease while older and more feeble participants would be likely to suffer severe illness and death. It’d be difficult to make an argument without contextualizing who would be most vulnerable to the disease and whether these vaccines really had any effect in them.

But issues involving these clinical trials don’t stop at issues with the primary endpoints, but also in how these studies were conducted.

For the most part, these trials can be boiled down to vaccinated participants with either the vaccine or a placebo, being set loose with researchers checking back and seeing if they tested positive via PCR or if they present with symptoms of COVID. Sounds decent, at least compared to other possible vaccine trials to say the least.

But one severe caveat to these studies is that, at the time they were conducted, we weren’t living in a normal society and conducting ourselves in the ways we used to. In late 2020 and early 2021 an overwhelming majority of the world was subject to non-pharmaceutical interventions⁵ via lockdowns, mask mandates, and extensive social isolation- all of which would actually run counter to the construction of a viable vaccine clinical trial that requires extensive socialization to obtain proper efficacy measures.

So how can one measure how well a vaccine protects a recipient when a majority of the world was stopped from seeing each other to an extensive degree?

It’d be rather hard to argue that a vaccine works when a large portion of the world was relegated to working from home (if at all) and made to order groceries online, and because of these circumstances there were no proper ways to test the actual effectiveness. It’s even more ironic considering that several of these clinical trials removed high-exposure participants even though they were the first to receive the vaccines (along with high-risk groups).

Here’s an excerpt from a 2020 CNBC article citing such demographics to first receive the vaccine:

Considering that we are working in hindsight, this also explains why these leaky vaccines may have been so “effective”. Remember that leaky vaccines are highly dependent upon incidences and exposure while AoN vaccines work independent of such factors. At the time that these vaccines were going through clinical trials, if the assumptions were made that they would work as AoN vaccines rather than leaky ones, then we could make the case that these vaccines would work irrespective of exposure risk and transmissibility of the virus.

And there was plenty of evidence to suggest that many in the medical industry assumed such a thing:

Most vaccine trials don’t properly simulate real-world exposures and risks, but these faults are likely to have been exacerbated further by NPIs that may have greatly altered the actual risks of COVID and have gone undetected due to limitations in statistical measurements and mathematical modeling. It's only through widespread administration and eventual lockdown removal did the real factors of a leaky vaccine come to a head, culminating into what would be considered a highly ineffective vaccine.

Things to consider

This post is intended to provide a different perspective with a good bit of pontification, focusing specifically on the ramifications of a leaky vaccine that was initially assumed to be a sterilizing, all-or-nothing vaccine. In no ways does it tread new territories, but at least provides additional context in the greater vaccine discourse.

Compared to all possible scenarios all-or-nothing vaccines would have be the best possible scenario if not for the adverse reactions. At the point that these vaccines were developed they were likely done under the assumption that they were sterilizing, and it was only through massive administration did we see that this was not the case.

As outlined above, the effectiveness of a leaky vaccine depends upon multiple factors, including exposure rate, transmission rate of the pathogen, and possible infectiousness of the vaccine recipient. The clinical trials were conducted during largescale NPI measures, which don’t represent the actual scope of normal social interactions and may have masked possible issues of vaccine leakiness during clinical trials.

Initial rollout of the vaccines also occurred a time in which many NPIs were still implemented, and it was only until vaccination received a high enough level that most areas eased off of these policies. At that point, it is likely that the full breath of leaky vaccine failures brought to light the true scope of these vaccine’s effectiveness. Greater infection rates after greater socialization left no more doubt the leakiness of these vaccines.

Once again, this post doesn’t provide anything substantially new, but at least explains what a leaky vaccine is, and how the widespread use of such a faulty product could lead to the fallout we are continuing to see to this day.

1. Edlefsen P. T. (2014). Leaky vaccines protect highly exposed recipients at a lower rate: implications for vaccine efficacy estimation and sieve analysis. Computational and mathematical methods in medicine, 2014, 813789. https://doi.org/10.1155/2014/813789

2. Ragonnet, R., Trauer, J. M., Denholm, J. T., Geard, N. L., Hellard, M., & McBryde, E. S. (2015). Vaccination Programs for Endemic Infections: Modelling Real versus Apparent Impacts of Vaccine and Infection Characteristics. Scientific reports, 5, 15468. https://doi.org/10.1038/srep15468

3. For all intents and purposes, when the x-axis variable changes we can assume that the other variables remain constant. Constant, baseline parameters include vaccine effectiveness of 70% (.7), vaccine coverage of 50% (.5), totally immunized vaccinees to be 0, and relative infectiousness of vaccinees to be 1. More can be found in Table 1.

4. Doshi P. Will covid-19 vaccines save lives? Current trials aren’t designed to tell us BMJ 2020; 371 :m4037 doi:10.1136/bmj.m4037

5. Note that this is not an actual endorsement of NPIs, but in order to remain consistent and provide context I am arguing that additional variables were likely to have factored into the vaccine’s supposed effectiveness which would have been considered in the data.