Estimation of Rotavirus Vaccine Effectiveness Based on Whole Genome Sequences

Reviewer #1 (Public review):

Summary:

Kwon et al present a very well-conducted and well-written sieve analysis of rotavirus infections in a passive surveillance network in the US, considering how relative vaccine efficacy changes with genetic distance from the vaccine strains including the whole genome. The results are compelling, supported by a number of sensitivity analyses, and the manuscript is generally easy to follow.

Strengths:

(1) The underlying study base, a surveillance network across multiple sites in the US.

(2) The use of a test-negative design, which is well established for rotavirus, to estimate vaccine efficacy.

(3) The use of genetic distance to measure differences between infecting and vaccine strains, and the innovative use of k-means clustering to make results more interpretable.

(4) The secondary and sensitivity analyses that provide additional context and support for the primary findings.

Weaknesses:

(1) As identified by the authors, there is a limited sample size for the analysis of RV1 (monovalent rotavirus vaccine).

(2) Sieve analyses were originally designed for randomized trials, in which setting their key assumptions are more likely to be met. There is little discussion in this paper of how those assumptions might be violated and what effect that might have on the results. The authors have access to some important confounders, but I believe some more discussion on potential biases in this observational study is warranted.

Reviewer #2 (Public review):

Summary:

This study introduces a new metric for assessing the efficacy of rotavirus vaccines through the genetic distance clustering of strains. The authors analyzed variations in vaccine protection using whole genome sequencing.

Strengths:

Evaluating vaccine efficacy using whole genome sequencing can enhance our understanding of how pathogen evolution influences disease transmission and control.

Weaknesses:

While the study proposed a new method for evaluating vaccine efficacy using genetic information, its weaknesses arise from the insufficient evidence that analyses based on whole genome sequencing are more reliable than those that rely solely on VP7 and VP4 genotypes.

Though most cases received the RV5 vaccine (n=119 compared to n=30 for RV1), Figure 2 and the primary focus of the paper concentrate on RV1, as the authors identified a stronger association with genetic distance.

Additionally, it is unclear whether the difference between the two groups (j=0 versus j=1) is statistically significant for the analysis based on genetic distance to the RV1 strain, as well as for that based on minimum genetic distance to any of the RV5 vaccine strains. In both cases, the confidence intervals show substantial overlap.

The authors do not seem to have used a criterion for model selection based on the number of clusters; therefore, k=2 may not represent the optimal number of clusters, particularly in relation to the genetic distance associated with the RV5 vaccine (Figure 1B), which does not appear to show a bimodal distribution.

Finally, outcomes for RV1 are highly associated with both homotypic and heterotypic antibody responses (Supplemental Figure 10), which have already been shown to impact vaccine effectiveness (The Pediatric Infectious Disease Journal 40(12):p 1135-1143, 2021, doi:10.1097/INF.0000000000003286). Given this strong association, the benefit of using genetic distance is unclear, as the GxPx genotype serves as a good proxy for genetic similarity.

Reviewer #3 (Public review):

Overall, this is an outstanding paper. It presents a novel approach to estimating rotavirus vaccine efficacy; is clearly written and presented; and has implications for this vaccine specifically as well as type-specific vaccine evaluation more generally. The analytical framework is a creative and there is rigorous use of data and statistical approaches. It has long been argued that rotavirus immunity/vaccine performance operates beyond the scale of G/P genotyping. This paper is the first to demonstrate that convincingly, using data on all 11 viral genes and whole genome sequence analysis. I have only minor comments that I recommend should be addressed.

Author response:

Public Reviews

Reviewer #1 (Public review):

Summary:

Kwon et al present a very well-conducted and well-written sieve analysis of rotavirus infections in a passive surveillance network in the US, considering how relative vaccine efficacy changes with genetic distance from the vaccine strains including the whole genome. The results are compelling, supported by a number of sensitivity analyses, and the manuscript is generally easy to follow.

Strengths:

(1) The underlying study base, a surveillance network across multiple sites in the US.

(2) The use of a test-negative design, which is well established for rotavirus, to estimate vaccine efficacy.

(3) The use of genetic distance to measure differences between infecting and vaccine strains, and the innovative use of k-means clustering to make results more interpretable.

(4) The secondary and sensitivity analyses that provide additional context and support for the primary findings.

Weaknesses:

(1) As identified by the authors, there is a limited sample size for the analysis of RV1 (monovalent rotavirus vaccine).

(2) Sieve analyses were originally designed for randomized trials, in which setting their key assumptions are more likely to be met. There is little discussion in this paper of how those assumptions might be violated and what effect that might have on the results. The authors have access to some important confounders, but I believe some more discussion on potential biases in this observational study is warranted.

We appreciate the reviewer’s positive comments and the opportunity to discuss the application of sieve analysis in observational vaccine effectiveness studies, contrasting it with its traditional use in clinical trials assessing vaccine efficacy. We fully acknowledge the reviewer's point that sieve analysis was originally developed for, and is most frequently employed in, randomized controlled trials (RCTs).

Sieve analysis, as defined by Gilbert et al. (2001), has the following core assumptions: (A1) uniform susceptibility to infection for all participants except for vaccine-induced strain-specific effects; (A2) equal exposure (for each strain s = 1,…,K ) distribution between vaccine groups; and (A3), constant strain prevalence. RCTs ensure these through randomization. However, our observational design is vulnerable to violating these assumptions, especially A1 and A3. To address A1 and A3, we adjusted for age (in years), sample collection year, and clinical setting (i.e., outpatient, inpatient, ED), aiming to account for both individual-level and temporal variations.

A2 is particularly challenging in observational settings. We found that study site was correlated with both vaccination status (main predictor) and the strain distribution, potentially violating A2. However, adjusting for study site reversed the expected association. Upon further reflection, we realized that the site-specific differences in strain distributions likely reflect the population-level effect of vaccination, which we believe outweighs the potential confounding by study site as an independent cause of both individual-level vaccination status and strain distributions irrespective of vaccination. Thus, adjusting for site would have obscured this genuine population-level effect, and therefore we elected not to do so. We will include further discussion of this point in the revised manuscript.

Our study demonstrates the unique capacity of sieve analysis to disentangle individual- and population-level effects on vaccine effectiveness in observational settings. We will expand on these considerations, including the potential biases inherent to observational studies and the rationale for our analytical choices, within the discussion section of the revised manuscript.

Reviewer #2 (Public review):

Summary:

This study introduces a new metric for assessing the efficacy of rotavirus vaccines through the genetic distance clustering of strains. The authors analyzed variations in vaccine protection using whole genome sequencing.

Strengths:

Evaluating vaccine efficacy using whole genome sequencing can enhance our understanding of how pathogen evolution influences disease transmission and control.

Weaknesses:

While the study proposed a new method for evaluating vaccine efficacy using genetic information, its weaknesses arise from the insufficient evidence that analyses based on whole genome sequencing are more reliable than those that rely solely on VP7 and VP4 genotypes.

Though most cases received the RV5 vaccine (n=119 compared to n=30 for RV1), Figure 2 and the primary focus of the paper concentrate on RV1, as the authors identified a stronger association with genetic distance.

Additionally, it is unclear whether the difference between the two groups (j=0 versus j=1) is statistically significant for the analysis based on genetic distance to the RV1 strain, as well as for that based on minimum genetic distance to any of the RV5 vaccine strains. In both cases, the confidence intervals show substantial overlap

The authors do not seem to have used a criterion for model selection based on the number of clusters; therefore, k=2 may not represent the optimal number of clusters, particularly in relation to the genetic distance associated with the RV5 vaccine (Figure 1B), which does not appear to show a bimodal distribution.

Finally, outcomes for RV1 are highly associated with both homotypic and heterotypic antibody responses (Supplemental Figure 10), which have already been shown to impact vaccine effectiveness (The Pediatric Infectious Disease Journal 40(12):p 1135-1143, 2021, doi:10.1097/INF.0000000000003286). Given this strong association, the benefit of using genetic distance is unclear, as the GxPx genotype serves as a good proxy for genetic similarity.

We sincerely appreciate reviewer's careful consideration of our manuscript and their constructive suggestions for improvement.

Regarding the comparison of whole-genome sequencing with traditional VP7/VP4 genotyping, we concur that a more explicit comparison would strengthen our findings. To this end, we plan to incorporate the direct comparison of genetic distance (GD) and genotype-specific vaccine effectiveness (VE) analyses into the main text. Additionally, we will conduct an analysis of VE based on homotypic, partially heterotypic, and fully heterotypic genotype groupings. This will provide a clearer demonstration of the potential added value of GD in refining VE estimates, particularly for future applications. Given the potential for reassortment among the rotavirus gene segments, our analysis highlights that relying solely on the VP7/VP4 genotype can at times be misleading.

Regarding k-means clustering, we wish to clarify that the selection of k=2 was not arbitrary. It was determined using the elbow method on the total within-sum-of-squares (using the fviz_nbclust function in the factoextra R package, with n=5000 bootstrapping). While we acknowledge that other methods, such as silhouette and gap statistics, may yield different optimal cluster numbers, we prioritized maximizing group sample sizes. We will explicitly state this model selection criterion within the methods section of the revised manuscript.

We acknowledge the reviewer’s concern regarding the overlapping confidence intervals and the statistical significance of the differences between the VE for the j=0 and j=1 groups. One way to address this would be to modify our analysis. Instead of two separate logistic regression models (controls vs j=0 cases, and controls vs j=1 cases), we could employ a multinomial logistic regression model with three categories: controls (reference), j=0 cases, and j=1 cases, then conduct Wald test to directly compare the regression slopes for the j=0 and j=1 cases against controls. We intend to explore this approach in the revised manuscript, which will provide a more rigorous assessment of differences in VE by accounting for the relationship between groups within a single model.

Reviewer #3 (Public review):

Overall, this is an outstanding paper. It presents a novel approach to estimating rotavirus vaccine efficacy; is clearly written and presented; and has implications for this vaccine specifically as well as type-specific vaccine evaluation more generally. The analytical framework is a creative and there is rigorous use of data and statistical approaches. It has long been argued that rotavirus immunity/vaccine performance operates beyond the scale of G/P genotyping. This paper is the first to demonstrate that convincingly, using data on all 11 viral genes and whole genome sequence analysis. I have only minor comments that I recommend should be addressed.

We sincerely thank the reviewer for their highly positive assessment of our manuscript. We will carefully address their minor comments and incorporate their recommendations in the revised manuscript, which we believe will further enhance the clarity and impact of our study.