The impact of pyrethroid resistance on the efficacy and effectiveness of bednets for malaria control in Africa
Abstract
Long lasting pyrethroid treated bednets are the most important tool for preventing malaria. Pyrethroid resistant Anopheline mosquitoes are now ubiquitous in Africa, though the public health impact remains unclear, impeding the deployment of more expensive nets. Metaanalyses of bioassay studies and experimental hut trials are used to characterise how pyrethroid resistance changes the efficacy of standard bednets, and those containing the synergist piperonyl butoxide (PBO), and assess its impact on malaria control. New bednets provide substantial personal protection until high levels of resistance, though protection may wane faster against more resistant mosquito populations as nets age. Transmission dynamics models indicate that even low levels of resistance would increase the incidence of malaria due to reduced mosquito mortality and lower overall community protection over the lifetime of the net. Switching to PBO bednets could avert up to 0.5 clinical cases per person per year in some resistance scenarios.
https://doi.org/10.7554/eLife.16090.001eLife digest
In recent years, widespread use of insecticidetreated bednets has prevented hundreds of thousands cases of malaria in Africa. Insecticidetreated bednets protect people in two ways: they provide a physical barrier that prevents the insects from biting and the insecticide kills mosquitos that come into contact with the net while trying to bite. Unfortunately, some mosquitoes in Africa are evolving so that they can survive contact with the insecticide currently used on bednets.
How this emerging insecticide resistance is changing the number of malaria infections in Africa is not yet clear and it is difficult for scientists to study. To help mitigate the effects of insecticide resistance, scientists are testing new strategies to boost the effects of bednets, such as adding a second chemical that makes the insecticide on bednets more deadly to mosquitoes. In some places, adding this second chemical makes the nets more effective, but in others it does not. Moreover, these doubly treated, or “combination”, nets are more expensive and so it can be hard for health officials to decide whether and where to use them.
Now, Churcher et al. have used computer modeling to help predict how insecticide resistance might change malaria infection rates and help determine when it makes sense to switch to the combination net. Insecticidetreated bednets provide good protection for individuals sleeping under them until relatively high levels of resistance are achieved, as measured using a simple test. As more resistant mosquitos survive encounters with the nets, the likelihood of being bitten before bed or while sleeping unprotected by a net increases. This is expected to increase malaria infections. As bednets age and are washed multiple times, they lose some of their insecticide and this problem becomes worse.
Churcher et al. also show that the combination bednets may provide some additional protection against resistant mosquitos and reduce the number of malaria infections in some cases. The experiments show a simple test could help health officials determine which type of net would be most beneficial. The experiments and the model Churcher et al. created also may help scientists studying how to prevent increased spread of malaria in communities where mosquitos are becoming resistant to insecticidetreated nets.
https://doi.org/10.7554/eLife.16090.002Introduction
It is estimated that 68% of the 663 million cases of malaria that have been prevented since the year 2000 have been through the use of longlasting insecticide treated bednets (LLINs) (Bhatt et al., 2015). However, there is a growing realisation that insecticide resistance is putting these advances under threat (WHO, 2012), with mosquitoes reporting widespread resistance to pyrethroids, the only class of insecticides currently approved for use in bednets (Ranson and Lissenden, 2016). The public health impact of pyrethroid resistance in areas of LLIN use is hard to quantify as a comparison between sites is complicated by multiple epidemiological factors making it difficult to ascribe differences in malaria metrics solely to mosquito susceptibility (Kleinschmidt et al., 2015). The efficacy of LLINs against mosquitoes is typically measured in experimental hut trials (WHO, 2013a). These experiments are time consuming, relatively expensive, and geographically limited and by themselves they do not fully account for all effects of the LLIN as they do not show the community impact (herd effects) caused by the insecticide killing mosquitoes (Killeen et al., 2007; Magesa et al., 1991). Mathematical models can be used to translate entomological endpoint trial data into predictions of public health impact. Currently this has only been done for a small number of sites (Briët et al., 2013) making it difficult for malaria control programmes to understand the problems caused by insecticide resistance in their epidemiological setting.
There are no easy to use genetic markers that can reliably predict the susceptibility of mosquitoes to pyrethroid insecticide (Weetman and Donnelly, 2015). The current most practical phenotypic method for assessing resistance is the use of bioassays which take wild mosquitoes and measures their mortality after exposure to a fixed dose of insecticide (WHO, 2013a). However the discriminating doses used in the assay are unrelated to the field exposure and so the predictive value of these bioassays for assessing the problems of pyrethroid resistance is unknown. A metaanalysis has shown that insecticide treated bednets still outperform untreated nets in experimental hut trials even against pyrethroid resistant populations (Strode et al., 2014) though the community impact (herd effects) of the LLIN was not assessed (Killeen et al., 2007). The population prevalence of pyrethroid resistance is known to be changing at a fast rate (Toé et al., 2014) making it important to regularly reevaluate the efficacy of LLINs in order to guide current vector control and resistance management strategies (WHO, 2012).
There are limited tools available for tackling pyrethroid resistance and protecting the advances made in malaria control. Until new LLINs containing alternative insecticide are available the only alternative bednet are those containing pyrethroids plus the insecticide synergist piperonyl butoxide (PBO). Studies have shown that PBO LLINs are substantially better at killing insecticide resistant mosquitoes in some locations but not others (Ngufor et al., 2014a, 2014b; Kitau et al., 2014; Asale et al., 2014; Ngufor et al., 2014c; Koudou et al., 2011; Corbel et al., 2010; Tungu et al., 2010; Malima et al., 2008; Adeogun et al., 2012a; Agossa et al., 2014; Malima et al., 2013). PBO LLINs are more expensive than standard LLINs, with one manufacturer’s 2012 price for PBO LLIN being US$4.90 compared to a comparable standard LLIN price of US$3.25 (Briët et al., 2013). This makes it unclear where and when their use would be beneficial over standard LLINs given constrained public health budgets. A mathematical modelling study used results from 6 experimental hut trials comparing a standard LLIN (PermaNet 2.0) with a PBO LLIN (PermaNet 3.0) against Anopheles gambiae sensu lato mosquitoes (Briët et al., 2013). It predicted that the more expensive PBO LLIN was still cost effective compared to a threshold of US$150/DALY averted (not comparing against standard LLINs) in 4 of the 6 sites, though these results are not generalisable beyond the specific sites chosen by the manufacturer, population prevalence of resistance, the type of LLIN or mosquito species. The WHO has recognised the increased bioefficacy of PermaNet 3.0 in some areas (WHO, 2015) but there is a lack of clear consensus on when and where these should be deployed. Defining the added public health benefit expected by a switch to PBO LLINs is essential to guide decisions on pricing, purchasing and deployment.
Here we propose that information on the current malaria endemicity, mosquito species and population prevalence of pyrethroid resistance (as measured by bioassay mortality) can be used to predict the public health impact of pyrethroid resistance and choosing the most appropriate LLIN for the epidemiological setting. Firstly (1) a metaanalysis and statistical model are used to determine whether mosquito mortality in a bioassay can be used to predict the proportion of mosquitoes, which die in experimental hut trials and to define the shape of this relationship. Secondly (2), another metaanalysis of experimental hut trial data is analysed to characterise the full impact of pyrethroid resistance on LLIN effectiveness. Thirdly, information from (1) and (2) is used to parameterise a widely used malaria transmission dynamics mathematical model to estimate the public health impact of pyrethroid resistance in different settings taking into account the community impact of LLINs. An illustration of model predictions showing how different malaria metrics change over time is given in Figure 1. The figure also indicates how LLIN coverage and variables such as malaria endemicity are incorporated in the model. Finally (4) this model is combined with bioassay and experimental hut trial results to predict the epidemiological impact of switching from mass distribution of standard to PBO LLIN.
Results
Defining a metric for pyrethroid resistance
The population prevalence of pyrethroid resistance is defined from the percentage of mosquitoes surviving a pyrethroid bioassay performed according to standardised methodologies. Data from all bioassay types (such as the WHO tube susceptibility bioassay (WHO, 2013b), WHO cone bioassay (WHO, 2013a) or CDC tube assay [Brogdon, 2010]) are combined to produce a simple to use generalisable metric. Note that this pyrethroid resistance test does not differentiate between varying levels of resistance within an individual mosquito as only single discriminating doses are used. It is assumed that the ability of a mosquito to survive insecticide exposure is not associated with any other behavioural or physiological change in the mosquito population which influences malaria transmission. For example, an increased propensity for mosquitoes to feed outdoors (subsequently referred to as behavioural resistance) would limit their exposure to LLINs though there is currently insufficient field evidence to justify its inclusion in the model (Briët and Chitnis, 2013; Gatton et al., 2013).
Using bioassays to predict LLIN efficacy
Table 1 summarises the datasets used in the different metaanalyses. Metaanalysis M1 shows that mosquito mortality in experimental hut trials can be predicted by the percentage of mosquitoes surviving a simple pyrethroid bioassay (Figure 2A). There is a substantial association between pyrethroid resistance in a bioassay and mortality measured in a standard LLIN experimental hut trial (Figure 2A, Deviance Information Criteria, DIC, with resistance as an explanatory variable = 2544.0, without = 2649.0 (lower value shows more parsimonious model), best fit parameters $\alpha}_{1$ = 0.634 (95% Credible Intervals, 95%CI, 0.012–1.29) and $\alpha}_{2$ = 3.99 [95%CI 3.171–5.12]). This indicates that bioassay survival can be used as a quantitative test to assess how the population prevalence of pyrethroid resistance influences LLIN efficacy. The number of studies identified in M1 is relatively small (only 21 datapoints) so the predictive ability of the bioassay was further validated using the A. gambiae s.l.PBO data (Figure 2B,C).

Figure 2—source data 1
 https://doi.org/10.7554/eLife.16090.008

Figure 2—source data 2
 https://doi.org/10.7554/eLife.16090.009

Figure 2—source data 3
 https://doi.org/10.7554/eLife.16090.010
Added benefit of PBO
The increased mortality observed by adding the synergist PBO to a pyrethroid bioassay was assessed for Anopheles funestus and Anopheles gambiae senu lato mosquitoes with different levels of pyrethroid resistance (M2, Figure 2B). Data suggests that for the A. gambiae complex PBO has the greatest benefit in mosquito populations with intermediate levels of pyrethroid resistance (including pyrethroid resistance as an explanatory variable DIC = 2544.0, without DIC = 4748.0). In A. funestus adding PBO appears to kill all mosquitoes irrespective of the prevalence of pyrethroid resistance (including resistance as an explanatory variable improved model fit, with DIC = 2544.0, without DIC = 2547.0, though the gradient of the line was so shallow as to effectively make the PBO synergised pyrethroid mortality independent of the population prevalence of pyrethroid resistance).
The relationships identified in Figure 2A and B are used to predict the added benefit of a PBO LLIN over a standard LLIN (Figure 2C). These predictions are consistent with the observed results from all published experimental hut trials directly comparing both LLIN types (M3) (see overlap of data points with model predictions on Figure 2C) providing further independent evidence that the population prevalence of pyrethroid resistance measured by a bioassay can be used to predict LLIN induced mortality in a hut trial for both standard and PBO LLINs.
The impact of pyrethroid resistance on LLIN efficacy
Mortality in experimental huts was shown to be a useful predictor of LLIN induced deterrence, exiting and the rate of pyrethroid decay (Figure 3A–C). Figure 3A indicates that the number of mosquitoes deterred from entering the experimental hut substantially decreases in areas of higher pyrethroid resistance (where LLIN induced mortality inside the hut is low) though the variability around the best fit line is high suggesting the precise shape of the relationship is uncertain. As the population prevalence of pyrethroid resistance increases (and mortality inside the hut decreases) an increasing proportion of mosquitoes entering the house exit without bloodfeeding (Figure 3B). Only when there is a very high population prevalence of pyrethroid resistance does the probability that a mosquito will successfully feed start to increase (Figure 3C). Changing behaviour of a host seeking mosquito with different levels of pyrethroid resistance is shown in Figure 3D.
The overall efficacy of an LLIN depends on its initial efficacy and the rate at which this changes over the lifetime of the net. Since there are currently no published durability studies in areas of high pyrethroid resistance or with PBO LLINs we estimate the loss of insecticidal activity from experimental hut trials using washed nets. Results indicate that washing decreases efficacy fastest in areas of higher pyrethroid resistance. Figure 3E shows estimates of the decay in pyrethroid activity assuming that the loss of efficacy due to washing is proportional to the change in activity seen over time (i.e. if the rate of decay over subsequent washes is twice as fast in a resistant mosquito population than the decay of pyrethroid activity over time will also be twice as fast). Mosquitoes with high pyrethroid resistance appear to overcome the insecticide activity of the LLIN faster than susceptible mosquitoes. A hypothesis for the cause of this relationship is outlined in Figure 3F.
The public health impact of pyrethroid resistance
The transmission dynamics model predicts that the higher the population prevalence of pyrethroid resistance the greater impact it will have on both the number of clinical cases (Figure 4A and B) and the force of infection (as measured by the EIR, Figure 4C). This is due to the lower initial killing efficacy of the LLIN but also because of the higher rate of decay of insecticidal activity (it gets less effective more quickly). The absolute increase in EIR caused by resistance increases in areas of high endemicity (Figure 4C), though the model predicts that the number of clinical cases caused will peak at intermediate parasite prevalence because high levels of clinical immunity will mask increased infection rates in hyperendemic areas. Understandably the impact of resistance will depend on the current LLIN coverage, with the total public health impact of resistance being greatest in areas where bednets were having the highest impact (i.e. areas of lower, 50%, coverage, see Figure 4—figure supplement 1). Equally the impact of resistance will be higher in areas with mosquito species which are more amenable to control through the use of LLINs (i.e. greater in Anopheles gambiae sensu stricto than Anopheles arabiensis, Figure 4—figure supplements 2 and 3). The transmission dynamics model predicts that the public health impact of pyrethroid resistance will be high. For example with as little as 30% resistance (70% mortality in discriminating dose assay) in a population with 10% slide prevalence (in 2–10 year olds) the model predicts that pyrethroid resistance would cause an additional 245 (95%CI 142–340) cases per 1000 people per year (Figure 4A, averaged over the 3 year lifeexpectancy of the net). Similar increases in the number of cases are seen in those with or without LLINs (Figure 4A).
The public health benefit of switching to PBO LLINs
The impact of the addition of the synergist, PBO, on pyrethroid induced mortality appears to depend on mosquito species and the population prevalence of pyrethroid resistance. In mosquito populations with moderate to high resistance results indicate PBO is an effective synergist of pyrethroids (Figure 5A). For example in an area with 10% endemicity and 80% resistance (20% mortality in discriminating dose assay) the model predicts that switching to PBO LLINs would avert an additional 501 (95%CI 319–621) cases per 1000 people per year (Figure 5A) compared to the same level of standard LLIN coverage. The absolute number of cases averted by switching to PBO LLINs is predicted to be greater in areas with intermediate endemicity as human immunity is likely to partially buffer the added benefit of PBO LLINs in areas of highest malaria prevalence (Figure 5B). However, due to the nonlinear relationship between incidence of clinical infection and endemicity, the greatest percentage reduction in clinical cases and EIR is seen in areas of low endemicity (Figure 5CF). The exact change in clinical cases will vary between settings. For example switching from 80% coverage with standard LLINs to 80% coverage with PBO LLINs in an area with 30% endemicity and a mosquito population with 60% pyrethroid resistance is predicted to reduce the number of clinical cases by ~60% whereas the same switch in the type of nets used in an area with 30% endemicity and 20% pyrethroid resistance would only reduce the number of clinical cases by ~20% (Figure 5C). Greater percentage reductions are likely to be seen in EIR than the number of clinical cases due to human immunity (Figure 5E).
Discussion
Pyrethroid resistance is widespread across Africa though its public health impact is unknown. Here we show that the simple bioassay can be used to predict how pyrethroid resistance is changing the efficacy of different types of LLIN and how this would be expected to influence malaria morbidity.
The bioassay is a crude tool for measuring pyrethroid resistance, though its simplicity makes it feasible to use on a programmatic level. Figure 2A and C indicate that on average bioassay mortality is able to predict the results of standard and PBO LLIN experimental hut trials for A. gambiae s.l. mosquitoes. There is a high level of measurement error in the bioassay (as seen by the wide variability in points in Figure 2A and B) so care should be taken when interpreting the results of single assays as differences in mosquito mortality may have been caused by chance. Multiple bioassays could be conducted on the same mosquito population and the results averaged to increase confidence. However the exact cause of the measurement error remains unknown so increased repetition many not necessarily generate substantially more accurate results as possible causes of variability, such as mosquito husbandry techniques or environmental conditions (Kleinschmidt et al., 2015), may be repeated. Further work is therefore needed to determine whether assay repetition substantially improves overall accuracy or whether further standardisation or more complex assays are required. The majority of data are for A. gambiae s.l. so the analysis needs to be repeated for other species once data becomes available. More advanced methods of measuring insecticide resistance (such as the intensity bioassay [Bagi et al., 2015] or the use of genetic markers [Weetman and Donnelly, 2015]) are likely to be a more precise way of predicting resistance. However, since there are insufficient data to repeat these analyses with these other assays their predictive ability remains untested. Similarly, this analysis has grouped WHO tube, WHO cone and CDC bottle assays together when the use of a single assay type might be more predictive.
The metaanalysis of experimental hut trials in areas with different levels of resistance has important implications for our understanding of how pyrethroid resistance influences LLIN efficacy. This analysis suggests that the probability that a mosquito will feed on someone beneath an LLIN only increases substantially at high levels of pyrethroid resistance (Figure 3C). People under bednets exposed to mosquito populations with intermediate levels of resistance still have a high degree of personal protection whilst in bed as those mosquitoes, which do not die are likely to exit the hut without feeding. It is only when mosquito populations are highly resistant (>60% survival) that an increasing proportion of mosquitoes appear to successfully feed through the LLIN (Figure 3D). This may explain why a previous metaanalysis on the impact of pyrethroid resistance on LLIN efficacy in experimental hut trials failed to find a substantial effect (Strode et al., 2014) as resistance was categorised into broad groups (partially based on highly variable bioassay data) unlike here where resistance is treated as a continuous variable (as measured using experimental hut trial mortality data which are less variable than bioassay data). This earlier study also only analysed papers published or presented prior to May 2013 and so it did not include the recent experimental hut trials which had the lowest mosquito mortality (Toé, 2015; Pennetier et al., 2013).
The metaanalysis revealed that the number of mosquitoes deterred from entering a hut with an LLIN, decreases with increasing pyrethroid resistance. LLIN efficacy is therefore reduced as mosquitos enter huts where they have both a higher chance of feeding and a lower chance of being killed. These parallel changes in behaviour increase the resilience of mosquito populations to LLINs as in a susceptible mosquito population, high deterrence will reduce LLIN efficacy by preventing mosquitoes entering houses where they have a high chance of being killed (relative to susceptible populations). Importantly the loss of deterrence suggests that those sleeping in a house with an LLIN though not sleeping under the net themselves (a phenomenon particularly common in older children [Nankabirwa et al., 2014]) will lose an additional degree of protection (on top of the community impact of mosquito killing).
The overall effectiveness of LLINs depends on the duration of insecticidal activity. Evidence suggests that multiply washed LLINs lose their ability to kill mosquitoes more in areas of high pyrethroid resistance. Washing is seen as an effective method of aging LLINs (WHO, 2013a). Repeatedly washing a net (and presumably reducing the concentration of the insecticide) appears to have little impact on its ability to kill a susceptible mosquito whilst significantly reducing the lethality of the LLIN against more resistant mosquitoes (Figure 2E). The difference in mortality is likely to be caused by mosquitoes with a higher population prevalence of resistance being able to tolerate a higher concentration of insecticide (WHO, 2013a). If so, then the higher longevity of LLINs against susceptible mosquitoes observed in the washed net data may be explained by the longer time it takes for the insecticide concentration on the LLIN to drop below this critical level (Figure 2F). This analysis assumes that the decay in pyrethroid activity over time is proportional to its decay following washing and this needs to be confirmed by durability studies in areas of high pyrethroid resistance. Nevertheless the results seem to be confirmed by two recent studies which evaluated mosquito mortality in older (standard) LLINs (Toé et al., 2014; Wanjala et al., 2015). Durability studies should be prioritised as the model predicts that, even at low levels of pyrethroid resistance, the loss of insecticide activity over the three year bednet lifeexpectancy, has a bigger epidemiological impact on malaria, than the initial efficacy of new LLINs. If confirmed then more regular net distribution could be considered as a temporary, albeit expensive, method to mitigate the public health impact of high pyrethroid resistance.
Transmission dynamics mathematical models are a useful tool for disentangling the different impacts of LLINs. Though a person under an LLIN requires high pyrethroid resistance before LLINs start to fail (Figure 3C), the models predict that at a population level even low pyrethroid resistance can increase the number of malaria cases over the lifetime of the net (Figure 4A). Hut trials measure feeding when the volunteer is underneath a bednet whilst in reality (and in the mathematical model) a percentage of mosquito bites are taken when people are not in bed. The loss of LLIN induced mosquito mortality is likely to decrease the community impact of LLINs, increasing average mosquito age and the likelihood that people are infected whilst unprotected by a bednet. This is primarily due to the shorter duration of insecticide potency of LLINs in mosquito populations with a higher prevalence of resistance (Wanjala et al., 2015). Without this change in the duration of pyrethroid activity, the epidemiological impact of pyrethroid resistance will only become evident once it reaches a high level (Figure 4A). The change in the community impact of LLINs can be seen in the increase in the number of cases in people who do not use nets. This change is substantial, reinforcing the need to consider community effects in any policy decision.
Detecting an epidemiological impact of a low population prevalence of resistance may be challenging for local health systems (for example, see < 20% resistance prevalence Figure 1—figure supplement 1, Figure 4) especially in an area where LLIN coverage, local climatic conditions and the use of other malaria control interventions are changing over time. These simulations also assume that resistance arrives overnight, when in reality it will spread through a mosquito population more gradually and therefore may be harder to detect. Mosquitoes exposed to LLINs may have reduced fitness (Viana et al., 2016). Currently the model assumes that mosquitoes which survive 24 hr after LLIN exposure are indistinguishable from unexposed mosquitoes. If this is not the case then hut trials data alone will be insufficient to predict the public health impact of pyrethroid resistance as current models will overestimate its impact. Similarly, if the mosquito population exhibits additional behavioural mechanisms to avoid LLINs, such as earlier biting times, in tandem to the increased tolerance of pyrethroid insecticide then the predictions presented here will likely underestimate the public health impact as this behaviour change has not been incorporated.
Currently a mosquito population is defined as being pyrethroid resistant if there is < 90% bioassay mortality (WHO, 2013b; Mnzava et al., 2015). Though useful, this entomological measure should not be considered as a measure of the effectiveness of pyrethroid LLINs. The personal protection provided by sleeping under an LLIN is likely to be substantial even at very high levels of resistance (Strode et al., 2014; Randriamaherijaona et al., 2015). Any reduction in mosquito mortality will likely reduce the community impact of LLINs though it may be hard to detect, especially in areas with new LLINs (the public health impact of resistance is likely to be greater in older nets, Figure 3E). As with all transmission dynamic mathematical models, these predictions need to be validated in particular locations with welldesigned studies combining epidemiological and entomological data. We are currently unaware of any published data with sufficient information to test the model against though a thorough validation exercise should be carried out as soon as such studies become available. Currently the metaanalyses and transmission dynamics models concentrated on malaria in Africa and give predictions for the three primary mosquito vector species found there. Each metaanalyses has data from multiple countries but these sites are not geographically representative of the whole of malaria endemic Africa. Though the principles outlined here may apply to other mosquito species in different care settings should be taken when extrapolating the results beyond the areas where the data were collated.
The bioassay data indicate that the ability of PBO to synergise pyrethroid induced mortality depends on the mosquito species. In A. funestus PBO always appears to restore near 100% mortality whilst for mosquitoes from the A. gambiae complex the greatest additional benefit of PBO being seen at intermediate levels of pyrethroid resistance (Figure 2B). The exact causes of this are unknown but is likely related to the predominant resistance mechanisms in each species. PBO’s primary synergistic effect of pyrethroids is thought to be due to the inhibition of the cytochrome P450 enzymes which catalyse the detoxification of the insecticides (Farnahm, 1998). Elevated P450 levels are the primary resistance mechanism in A. funestus whereas in A. gambiae s.l. both increased detoxification and alterations in the target site contribute to pyrethroid resistance with the latter mechanism being largely unaffected by PBO (Mulamba et al., 2014; Riveron et al., 2013).
For A. gambiae s.l. populations this result was verified by experimental hut trial data which directly compare standard and PBO LLINs (Figure 2C). Both bioassay and hut trial data suggest a minimal additional benefit of PBO in areas with very high levels of pyrethroid resistance. Unfortunately, there are currently no published studies where PBO LLINs have been tested in experimental hut trials in areas with A. funestus so these bioassay results should be treated with caution until they can be further verified. Additional data would also allow the differences between species in the A. gambiae complex to be assessed. A previous analysis comparing PermaNet 2.0 and 3.0 was unable to test whether the increase in efficacy of the PBO LLIN was solely due to the addition of PBO as this net has a higher concentration of insecticide (Briët et al., 2013). The results presented here show a consistent pattern between PermaNet 2.0 and 3.0 and Olyset and Olyset Plus. As both Olyset nets have the same concentration of insecticide, this suggests that PBO is causing the enhancement of efficacy.
The WHO recommends that countries routinely conduct nonPBO pyrethroid bioassays as part of their insecticide resistance management plan (WHO, 2012). In areas with A. gambiae s.l. the evidence presented here suggests that the results of bioassays with and without PBO can be used to predict the additional public health benefit of PBO LLINs. If there is a greater mortality in the PBO bioassay and the relative mortalities broadly agree with the red curve in Figure 2B, then Figure 5B can be used to predict the approximate number of cases that will be saved by switching from standard to PBO LLINs (for a given level of endemicity and LLIN coverage). Areas with 40–90% survival (10–60% mortality) in a nonPBO standard bioassay (of any type) should consider conducting PBO synergism bioassays to determine the suitability of PBO LLINs. We would suggest that either the WHO cone, WHO tube or CDC bottle assay (conducted in triplicate and averaged to improve precision) should be sufficient evidence to justify the need to switch to PBO LLINs.
The decision to recommend PBO nets over standard LLINs requires information on the relative cost effectiveness and affordability of PBO nets. If both net types cost the same and resistance has been detected then this work indicates that PBO LLINs should always be deployed as evidence suggests that they are always more effective. However, if PBO nets are more expensive, then cost effectiveness analysis will be required. The results of such analysis are likely to be context specific (depending on price, resistance level, endemicity and coverage) and interpreting them will require information on decision makers’ willingness and ability to pay for additional effectiveness. In many situations, malaria control budgets are likely to be fixed and therefore switching to more expensive PBO LLINs may cause a reduction in overall bednet coverage. The impact of reduced coverage must therefore be off set against the benefits of introducing PBO nets, taking into consideration any additional factors such as changed programmatic costs, and equity issues.
Rapid deployment of new vector control products saves lives and gives incentives for industry to invest in new methods of vector control. New methods are likely to have a higher unit price than existing tools so it is important to be able to determine where and when they should be deployed in an efficient and transparent manner. Frameworks such as those presented here offer a relatively straightforward method of comparing two different products to determine whether the increased effectiveness justifies the higher unit price.
Much of the debate over the impact of pyrethroid resistance on LLIN effectiveness has focused on the loss of personal protection provided by new nets and does not fully take into account their community impact. A large body of evidence has shown how widespread use of LLINs can cause considerable community protection, both to those who use bednets and nonusers (Killeen, 2007 and references therein). Therefore the community impact should be considered in any study investigating the consequences of pyrethroid resistance (Briët et al., 2013; Killeen, 2014), as any reduction in mosquito killing is likely to increase malaria cases even in areas with mildly resistant mosquito populations where LLINs are still providing good personal protection. The evidence presented here suggests that high levels of pyrethroid resistance are likely to have a bigger public health impact than previously thought and therefore could represent a major threat to malaria control in Africa.
Materials and methods
Description of data
Request a detailed protocolTo generate results which are broadly applicable to all mathematical models were fit to data compiled by systematic metaanalyses of the published literature. Where possible metaanalyses were extended to the grey literature by including unpublished information. These include unpublished bioassay data from Liverpool School of Tropical Medicine, submissions to the World Health Organisation Pesticide Evaluation Scheme (WHOPES) and results from unpublished experimental hut trials (collated by contacting LLIN manufacturers VestergaardFrandsen and Sumitomo Chemicals Ltd). The metaanalyses followed the Preferred Reporting Items for Systematic Reviews and MetaAnalyses guidelines (Moher et al., 2009) for study search, selection and inclusion criteria though the study was not registered. The predefined inclusion criteria of each of the metaanalyses are presented in Table 2 whilst the predefined search strings and the databases searched are outlined in full in Figure 2—source data 1. Extraction was done by N.L. into piloted forms. Study corresponding authors were contacted for raw data when this information was unavailable (all contacted investigators responded with the requisite information).
Impact of pyrethroid resistance on LLIN mortality
Request a detailed protocolTo determine whether simple pyrethroid bioassays can be used to infer the outcome of experimental LLIN hut trials a metaanalysis (summarised as Metaanalysis 1, M1) was conducted to identify studies where both were carried out concurrently. To test whether this relationship changed with the population prevalence of insecticide resistance simple functional forms were fit to the raw data using a mixedeffect logistic regression (summarised as Relationship 1, R1). There has been an attempt to standardise bioassay and experimental hut trial procedures to enable data from different studies to be directly compared. These include using standard concentrations of insecticide, mosquito exposure time and mosquito husbandry in bioassays, hut design, trap type and the use of human baits in experimental hut trials. Nevertheless, some procedural discrepancies remain between studies, for example, in bioassays the age and sex of mosquitoes and how they were collected (e.g. F1 progeny of wild caught mosquitoes or wild caught larvae reared in insectary and tested as adults). These covariates and others (for example information on genetic markers associated with insecticide resistance), could be included within the analysis, though their addition would increase data needs of future studies and complicate the use of study results. Instead a mixedeffects binomial regression is adopted which allows mosquito mortality to vary at random between studies. This statistical method enables a wider selection of studies to be included within the analysis, produces more generalizable results and reduces problems caused by data autocorrelation. Mosquito mortality in an experimental hut trial is defined as the proportion of mosquitoes, which enter the hut which die, either within the hut or within the next 24 hr.
Metaanalysis 1 (M1) identified only 7 studies where concurrent bioassays and experimental hut trials were carried out (Table 3). Given the paucity of data results from all types of bioassay and mosquito species were combined and a simple, functional form was used to describe the relationship (the fixedeffect). Let $x$ denote the proportion of mosquitoes dying in a standard (nonPBO) pyrethroid bioassay then the population prevalence of pyrethroid resistance (expressed as a percentage, denoted $I$) is described by the following equation,
Extending the notation of Griffin et al. (2010) the proportion of mosquitoes, which died in a hut trial is denoted ${l}_{p}$, where subscript $p$ indicates the net type under investigation, be it a nonet control hut ($\mathit{p}=0$), a standard nonPBO LLIN ($\mathit{p}=1$), or a PBO LLIN ($\mathit{p}=2$). For a standard LLIN it is assumed to be explained by the equation,
where parameters $\alpha}_{1$ and $\alpha}_{2$ define the shape of the relationship and $\tau$ is a constant used to centre data to aid the fitting process. More sophisticated functional forms could be used for R1 (Equation [2]) though they were not currently warranted given the limited dataset. Let $N}_{p$ indicate the number of mosquitoes entering a hut in an experimental hut trial. If the number of these mosquitoes which enter the hut and subsequently die ($L}_{1$) follows a binomial distribution then parameters ${\alpha}_{1}$ and ${\alpha}_{2}$ can be estimated for a nonPBO net by fitting the following equation to M1,
The randomeffects component is included by allowing mortality to vary at random between sites by adding the error term $\u03f5}_{\alpha$ which has a mean of zero and a constant variance.
Estimating the impact of PBO on pyrethroid induced mortality
Request a detailed protocolThe number of experimental hut trials investigating the difference between standard and PBO nets is limited. Instead a metaanalysis of all bioassay data investigating the impact of PBO on pyrethroid induced mosquito mortality is undertaken incorporating all published and unpublished literature (M2, Table 4). Bioassay mortality can be influenced by a multitude of factors including assay type, temperature and relative humidity (Kleinschmidt et al., 2015). To account for this difference between studies, the relationship between the benefit of adding PBO and the population prevalence of pyrethroid resistance was estimated using a mixedeffect logistic regression (R2). Preliminary analysis suggests that the shape of the relationship is relatively complex and cannot simply be described by the use of a standard linear function typically used in regression. Since the added benefit of PBO in a given population will ultimately be determined by the shape of this relationship a variety of different functional forms are tested statistically. It was initially intended to include the type of assay used (e.g. WHO tube assay, WHO cone assay or CDC bottle assay) as an additional fixed effect, though the paucity of data (especially comparing bioassay mortality to experimental hut trial mortality) meant that data from all assays were combined and this covariate was excluded. As the same type of assay are used for both nonPBO and PBO tests this should not bias the results and will generate recommendations that are generalizable across all three assay types. The proportion of mosquitoes killed by pyrethroid insecticide in a bioassay with the addition of PBO is denoted $f$ and is given by the equation:
where $x$ is the proportion of mosquitoes dying in a nonPBO bioassay, parameters, $\beta}_{1},\phantom{\rule{thinmathspace}{0ex}}{\beta}_{2$ and $\beta}_{3$ define the shape of the relationship and $\tau $ is a constant supporting the fitting process (this relationship is referred to as R2). Let $A}_{i$ be the number of mosquitoes used in a bioassay and ${D}_{i}$ the number which died, with subscript $i$ denotes whether or not PBO was added to the bioassay ($i$ = 1 pyrethroid alone, $i$ = 2 pyrethroid plus PBO). If it is assumed that the number of mosquitoes that die in the bioassay follows a binomial distribution then parameters, $\beta}_{1},\phantom{\rule{thinmathspace}{0ex}}{\beta}_{2$ and $\beta}_{3$ can be estimated by fitting the following equations to the dataset from (M1),
Parameter $\u03f5}_{\beta$ represents a normally distributed random error with a mean of zero and a constant variance and is estimated from the fitting procedure.
Predicting the added benefit of PBO LLINs in experimental hut trials
Request a detailed protocolRelationships R1 and R2 can be used to predict the effectiveness of PBO LLINs in experimental hut trials. When bioassay data are unavailable the current population prevalence of insecticide resistance can be predicted from mosquito mortality measured in a standard LLIN experimental hut trial by rearranging Equation [2],
where the section in round brackets is the inverse logit function. This equation together with Equations [2] and [4] can be then used to predict the relationship between hut trial mortality in standard and PBO LLINs for a range of areas with different levels of pyrethroid resistance using the following steps (a) to (c) below.
For a range of values of $l}_{1$ (proportion of mosquitoes which died in a standard LLIN hut trial) generate the predicted population prevalence of mosquito mortality in a bioassay expected in the population $\hat{x}$ using Equation [7].
Use $\hat{x}$ to predict pyrethroid induced mortality in a bioassay with PBO $\hat{f}$ given the current population prevalence of pyrethroid resistance (i.e. substitute $\hat{x}$ for $x$ in Equation [4]).
Convert the expected mortality in a bioassay $\hat{f}$ into the expected mortality in a PBO LLIN hut trial (i.e. substitute $\hat{f}$ for $\hat{x}$ in Equation [2]).
To test the predictive ability of R1 and R2 a third metaanalysis was carried out for all experimental hut trials which directly compare standard and PBO pyrethroid LLINs (M3, Table 5). The accuracy of these predictions can then be examined by comparing them visually (Figure 2C) or statistically using an Anaylsis of Variance.
Quantifying the impact of standard and PBO LLINs in the presence of insecticide resistance
Request a detailed protocolThe impact of insecticide resistance on mosquito interactions with LLINs is systematically investigated by analysing the experimental hut trials identified in M3. Restricting the analysis to the two most commonly used standard LLINs minimises the interstudy variability and allows the different behaviours of mosquitoes exposed to standard and PBO LLINs to be directly assessed. Following a widely used transmission dynamics model of malaria (Griffin et al., 2010; Walker et al., 2015) it is assumed that an LLIN can alter a hostseeking mosquito behaviour in one of three ways: firstly it can deter a mosquito from entering a hut (an exitorepellency effect); secondly the mosquito can exit the hut without taking a bloodmeal; and thirdly it could kill a mosquito (with the mosquito either being fed or unfed). A mosquito that isn’t deterred, exited or killed will successfully bloodfeed and survive. The public health benefit of LLINs depends not only on their initial effectiveness but also on how the properties of the net changes over its lifetime. The ability of a net to kill a mosquito will decrease over time as the quantity of insecticide active ingredient declines. The nonlethal protection provided by the LLIN may also decrease with the decay of the active ingredient and the physical degradation of the net (i.e. the acquisition of holes). It is assumed that the underlying difference in hut trial mortality between sites for standard LLINs is caused by the mosquito population having a different population prevalence of pyrethroid resistance. Pyrethroid resistance may also influence the relative strength of LLIN deterrence and exiting and it is important to characterise these modifications of behaviour as they contribute substantially to the population level impact of mass LLIN distribution. Visual inspection of these data indicates that mosquito deterrence and exiting can be described by the degree of mosquito mortality seen in the same hut trial.
The proportion of mosquitoes not deterred from entering a hut by the LLIN is estimated using ${m}_{p}$, the ratio of the number of mosquitoes entering a hut with an LLIN (${N}_{1}$or ${N}_{2}$) to the number entering a hut without a bednet (${N}_{0}$, here assumed to be the same as a hut with an untreated bed net). A statistical model is used to determine whether there is an association between the number of mosquitoes entering a hut with a standard LLIN and the proportion of mosquitoes which die when they do (which is assumed to be a proxy for mosquito susceptibility, i.e. ${m}_{1}$ is described by ${l}_{1}$ and ${m}_{2}$ is described by ${l}_{2}$). It is assumed that the shape of the relationship between the proportion of mosquitoes entering a hut with an LLIN relative to a hut with an untreated net (1deterrence) and mortality is described by the flexible third order polynomial,$$
Though there is no a priori reason to assume an inflection point in the relationship between ${m}_{p}$ and ${l}_{p}$ the polynomial function is chosen as it is highly flexible and would allow such a curve should it exist (which is necessary given the variability in the raw data). The shape parameters ${\delta}_{1}$, ${\delta}_{2}$ and ${\delta}_{3}$ are estimated assuming that the number of mosquitoes caught has a normal distribution (verified using a and deterrence is allowed to vary at random between sites (with variance ${\delta}_{4}$).
The proportion of mosquitoes entering the hut which exit without feeding is denoted ${j}_{p}$ whilst the proportion which successfully feed upon entering is ${k}_{p}$. Once entered the hut mosquitoes have to either exit, die or successfully feed (i.e. $1={j}_{p}+{l}_{p}+{k}_{p}$). Visual inspection of these data indicates that ${k}_{p}$ increases with decreasing mortality at an exponential rate (Figure 3C). Therefore, if the number of mosquitoes which feed and survive (${S}_{p}$) follows a binomial distribution then,
where ${\theta}_{1}$ and ${\theta}_{2}$ determine the shape of the relationship and ${\u03f5}_{\theta}$ is a normally distributed random error which varies between sites.
Parameterising transmission dynamics model
Request a detailed protocolEstimates of${j}_{p}$, ${l}_{p}$ and ${m}_{p}$ can be used to determine the proportion of mosquitoes repeating (a combination of deterrence and exiting, ${r}_{p0}$), dying (${d}_{p0}$) and feeding successfully (${s}_{p0}$) during a single feeding attempt in a hut with a new LLIN relative to those successfully feeding in a hut without an LLIN (i.e. $p$=1 or 2),
where ${j}_{p}=1{l}_{p}{k}_{p}$, ${j}_{p}^{\text{'}}={m}_{p}{j}_{p}+\left(1{m}_{p}\right)$, ${k}_{p}^{\text{'}}={m}_{p}{k}_{p}$ and ${l}_{p}^{\text{'}}={m}_{p}{l}_{p}$ (Griffin et al., 2010). Not all mosquitoes which enter a house will successfully feed even if there are no vector control interventions inside. The experimental hut trials used in this analysis did not include a nonet control (${k}_{0}$) so historical studies are used for this parameter (Curtis et al., 1996; Lines et al., 1987). Though theoretically ${s}_{p0}$ could have values > 1 for practical purposes, it is constrained between zero and one as on average mosquitoes entering a hut with an LLIN are less likely to feed than a mosquito entering a hut without a bednet (as shown by all estimates of ${k}_{p}$ being substantially lower than ${k}_{0}$, see Figure 3C and Table 6). The majority of experimental hut trials in M3 are in areas where the dominant vector is A. gambiae s.s. and no studies were conducted in areas with A. funestus. As there is insufficient information to generate these functions for each species separately it is assumed that the relationship between ${r}_{p0}$, ${s}_{p0}$ and ${d}_{p0}$ are consistent across all species. The average effectiveness of LLINs in an entirely susceptible mosquito population identified in M3 is slightly higher than those analysed by Griffin et al. (2010) which included a wider range of LLIN types. Values of ${m}_{p}$ (the propensity of mosquitoes to enter a hut with an LLIN relative to one without) greater than one are truncated at one as there is insufficient evidence to justify that mosquitoes preferentially enter huts with LLINs (in part because the number of studies with very low mortality are low and the metric has high measurement error).
Decay in LLIN efficacy over time
Request a detailed protocolThe ability of a net to kill a mosquito will decrease over time as the quantity of insecticide active ingredient declines. The nonlethal protection provided by the LLIN may also decrease with the decay of the active ingredient and the physical degradation of the net (i.e. the acquisition of holes). To fully capture the loss of efficacy of an LLIN requires a net durability survey to be carried out over multiple years. To our knowledge, no durability studies have been published in areas of high pyrethroid resistance nor using the new generation of LLINs with the addition of PBO. In the absence of these data, we use the results from experimental hut trials that washed the net prior to its use. These experimental huts give some indication of how mosquitoes react to the change in insecticide concentration, though they do not provide information on the physical durability of the net (as holes in the net are artificially generated). For simplicity and following (Griffin et al., 2010) it is assumed that the killing activity of pyrethroid over time (the halflife in years, denoted ${H}_{y}$) is proportional to the loss of morbidity caused by washing (the halflife in washes, ${H}_{w}$). A prior estimate of the halflife in years (Mahama et al., 2007) from a durability study of a nonPBO LLIN with susceptible mosquitoes (${H}_{y}^{s}$) is then used to reflect changes caused by pyrethroid resistance by,
where superscript $s$ indicates the halflife in a fully susceptible mosquito population (i.e. ${l}_{1}$ = 1). Note that if the newer PBO nets have better durability than standard LLINs then this will under estimate their additional benefit. Following Griffin et al., 2010 it is assumed that the activity of the insecticide decays at a constant rate according to a decay parameter ${\gamma}_{p}$, which is related to the halflife by ${H}_{w}=\text{ln}\left(2\right)/{\gamma}_{p}$. To test whether the rate of decay changes with ${l}_{p}$ (i.e. mosquito mortality caused by new standard and PBO LLINs) the following equation was fit to M3,
Shape parameters ${\mu}_{p}$ and ${\rho}_{p}$ are allowed to vary between net types. The proportion of mosquitoes repeating due to the LLIN decreases from a maximum, ${r}_{p0}$, to a nonzero level ${r}_{M}$, reflecting the protection still provided by an LLIN that no longer has any insecticidal activity. For simplicity, it is assumed that the rate of decay from ${r}_{p0}$ to ${r}_{M}$ is given by ${\gamma}_{p}$ (as the degradation of the net over time is unlikely to be recreated by washing). The full equations for the proportion of mosquitoes repeating, dying and successfully feeding at time $t$ following LLIN distribution (${r}_{p}$, ${d}_{p}$ and ${s}_{p}$, respectively) is given by,
Fitting procedure
Request a detailed protocolAll models were fit using a Markov chain Monte Carlo sampling algorithm implemented in the programme OPENBUGS (Lunn et al., 2009). This Bayesian method enabled measurement error to be incorporated in both the dependent and independent variables according to the number of mosquitoes sampled (both in bioassays and hut trials). Uninformative priors were used for all parameters with the exception of the random effects variance parameters which were constrained to be positive (though were still uninformative,see Source code 1 in the Supplementary Information for a full list of priors). Three Markov chains were initialized to assess convergence and the first 5000 Markov chain Monte Carlo iterations were discarded as burn in. Convergence was assessed visually and a total of 10,000 iterations were used to derive the posterior distribution for all parameters and to generate 95% Bayesian credible interval estimates for model fits. The models were compared using the deviance information criterion (DIC) where the smaller value indicates a better fit, and a difference of five deviance information criterion units is considered to be substantial (Spiegelhalter et al., 2002). Equations [8] to [19] were fit simultaneously to M3 enable the impact of washed nets to contribute to the relationship between ${r}_{p}$, ${d}_{p}$ and ${s}_{p}$, through the decay function, ${\gamma}_{p}$, doubling the number of datapoints in the analysis. A direct comparison between net types is beyond the scope of this study. Only one study compared PermaNet 2.0 and PermaNet 3.0 at the same time and place as Olyset and Olyset Plus and this study did not conduct hut trials with washed LLINs. As the different nets were tested in areas with different levels of pyrethroid resistance (in part because the low overall number of studies) then the impact of resistance and net type cannot currently be disentangled.
Predicting the public health impact of insecticide resistance
Request a detailed protocolThe public health benefit of PBOLLINs will depend on the epidemiological setting in which they are deployed. This includes the baseline characteristics of the setting (e.g. mosquito species, abundance and seasonality), history of malaria control interventions (e.g. prior use of bednets, management of clinical cases) and prevalence of insecticide resistance. The rate at which pyrethroid resistance has evolved is highly uncertain. It is likely that it first became evident through its use in agriculture and the relative contribution of vector control to the selection of resistance is unknown and will vary between sites. This makes it impossible to recreate the spread of resistant phenotypes in a particular setting and predict its cumulative public health impact without detailed longitudinal studies spanning decades (which do not exist for malaria endemic regions). Instead the impact of pyrethroid resistance is estimated by assuming it arrives instantaneously at a given level. To generate a broadly realistic history of LLIN usage it is assumed that LLINs were introduced at a defined coverage at year zero and redistributed every three years to the same percentage of the human population (Figure 1). The mosquito population is assumed to be either A. gambiae s.s., A. arabiensis or Anopheles funestus (the three major vectors in Africa) which are entirely susceptible to pyrethroids up until year 6 when pyrethroid resistance arrives instantaneously. The public health impact of resistance is then measured over the subsequent three years (the average clinical incidence or entomological inoculation rate (EIR) between the years 6 and 9) and compared to a population where resistance did not arise. The impact of PBO LLINs is predicted by introducing them into the resistant population at the year 9 and then measuring over the subsequent 3 years. For simplicity, it is assumed that there is perennial transmission, no other type of vector control and that once introduced pyrethroid resistance remains constant. Though perennial transmission is unrealistic it is necessary in order to produce simple guidelines (as there is a very high number of combinations of seasonal patterns, relative mosquito species abundance and timings of LLIN distribution campaigns). A sensitivity analysis with more realistic seasonal patterns shows the change in clinical incidence compared to the perennial transmission is relatively minor, in part because the LLINs are used over 3 yearly cycles and their decay in effectiveness is relatively slow. LLINs are initially distributed at time zero at random (i.e. there was no targeting to those with the highest infection) and from then on the same people receive them every campaign to ensure that coverage remains at the defined level (i.e. the number of people with an LLIN would go up if the distribution was random each round). Realistic usage patterns are adopted to reflect higher coverage immediately after LLIN distribution. No other vector control is incorporated whilst 35% of clinical cases are assumed to receive treatment, 36% which receive an ACT (estimated by averaging across Africa using data collated by Cohen et al., [2012]). A full list of the parameters, their definitions and estimated values are given in Table 6 whilst all other parameters are taken from Griffin et al. (2010) and White et al. (2011).
To investigate how the uncertainty in mosquito behaviour and the impact of PBO influence model predictions, a full sensitivity analysis is carried out for the parameters determining LLIN efficacy. A thousand parameter sets for ${\alpha}_{1}$, ${\alpha}_{2}$, ${\beta}_{1},{\beta}_{2},$, ${\delta}_{1}$, ${\delta}_{2}$, ${\theta}_{1}$, ${\theta}_{2}$,$\text{}{\mu}_{p}$ and $\text{}{\rho}_{p}$ are sampled from the posterior distribution and are used to generate a range of possible values for ${r}_{p0}$, ${s}_{p0}$, ${d}_{p0}$ and $\text{}{\gamma}_{p}$ (Figure 4—figure supplement 5). This allows uncertainty in all measurements (such as the relationship between resistance and hut trial mortality) to be propagated throughout the equations. These parameter sets are then included as runs within the full transmission dynamics model to unsure the full uncertainty in these data is represented and the 95% credible intervals for model outputs are then shown.
Source data
Request a detailed protocolFigure 2—source data 1–3. Figure 2—source data 4 is hosted on Dryad (doi:10.5061/dryad.13qj2)
Data availability

Data from: The impact of pyrethroid resistance on the efficacy and effectiveness of bednets for malaria control in AfricaAvailable at Dryad Digital Repository under a CC0 Public Domain Dedication.
References

Villagescale evaluation of PermaNet 3.0: an enhanced efficacy combination longlasting insecticidal net against resistant populations of Anopheles gambiae s.sMalaria Chemotherapy, Control and Elimination 1:1–9.https://doi.org/10.4303/mcce/235543

Efficacy of a combination long lasting insecticidal net (PermaNet® 3.0) against pyrethroid resistant Anopheles gambiae s.s. and Culex quinquefasciatus: an experimental hut trial in NigeriaNigerian Journal of Clinical & Biomedical Research 6:37–50.

Evidence of multiple pyrethroid resistance mechanisms in the malaria vector Anopheles gambiae sensu stricto from NigeriaTransactions of the Royal Society of Tropical Medicine and Hygiene 103:1139–1145.https://doi.org/10.1016/j.trstmh.2008.08.021

Guideline for evaluating insecticide resistance in vectors using the CDC bottle bioasssay 2010 with inserts 1 (2012) and 2 (2014)

Bioassay and biochemical analyses of insecticide resistance in southern African Anopheles funestus (Diptera: Culicidae)Bulletin of Entomological Research 91:265–272.https://doi.org/10.1079/BER2001108

Comparison of different insecticides and fabrics for antimosquito bednets and curtainsMedical and Veterinary Entomology 10:1–11.https://doi.org/10.1111/j.13652915.1996.tb00075.x

Piperonyl Butoxide: The Insecticide SynergistChapter 12: The mode of action of piperonyl butoxide with reference to studying pesticide resistance, Piperonyl Butoxide: The Insecticide Synergist, Acadmeic Press.

Efficacy of PermaNet® 3.0, Olyset® Plus and Yorkool® against natural populations of An. gambiae s.l. in experimental huts, Kolokopé, Togo

The BUGS project: Evolution, critique and future directionsStatistics in Medicine 28:3049–3067.https://doi.org/10.1002/sim.3680

Preferred reporting items for systematic reviews and metaanalyses: the PRISMA statementAnnals of Internal Medicine 151:264–269.https://doi.org/10.7326/00034819151420090818000135

An experimental hut evaluation of PermaNet® 3.0, a deltamethrin–piperonyl butoxide combination net, against pyrethroidresistant Anopheles gambiae and Culex quinquefasciatus mosquitoes in southern BeninTransactions of the Royal Society of Tropical Medicine and Hygiene 104:758–765.https://doi.org/10.1016/j.trstmh.2010.08.008

Malaria in schoolage children in Africa: an increasingly important challengeTropical Medicine & International Health 19:1294–1309.https://doi.org/10.1111/tmi.12374

Combining organophosphatetreated wall linings and longlasting insecticidal nets fails to provide additional control over longlasting insecticidal nets alone against multiple insecticideresistant Anopheles gambiae in Côte d'Ivoire: an experimental hut trialMalaria Journal 13:396.https://doi.org/10.1186/1475287513396

An experimental hut evaluation of PermaNet(®) 3.0, a deltamethrinpiperonyl butoxide combination net, against pyrethroidresistant Anopheles gambiae and Culex quinquefasciatus mosquitoes in southern BeninTransactions of the Royal Society of Tropical Medicine and Hygiene 104:758–765.https://doi.org/10.1016/j.trstmh.2010.08.008

Bayesian measures of model complexity and fitJournal of the Royal Statistical Society: Series B 64:583–639.https://doi.org/10.1111/14679868.00353

Characterisation of insecticide resistance in Anopheles gambiae from Burkina Faso and its impact on current malaria control strategiesPhD thesis, University of Liverpool.

Increased pyrethroid resistance in malaria vectors and decreased bed net effectiveness, Burkina FasoEmerging Infectious Diseases 20:1691–1696.https://doi.org/10.3201/eid2010.140619

Evolution of insecticide resistance diagnostics in malaria vectorsTransactions of the Royal Society of Tropical Medicine and Hygiene 109:291–293.https://doi.org/10.1093/trstmh/trv017

Global plan for insecticide resistance management in malaria vectors (GPIRM)

BookGuidelines for Laboratory and Field Testing of LongLasting Insecticidal NetsGeneva: World Health Organisation.

BookTest Procedures for Insecticide Resistance Monitoring in Malaria Vector MosquitoesGeneva: World Health Organisation.

Report on the second meeting of the WHO vector control advisory group. WHO/HTM/NTD/VEM/2014.2
Decision letter

Simon I HayReviewing Editor; Institute for Health Metrics and Evaluation, United States
In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.
Thank you for resubmitting your work entitled "The impact of pyrethroid resistance on the efficacy and effectiveness of bednets for malaria control" for further consideration at eLife. Your revised article has been favorably evaluated by Prabhat Jha (Senior editor), a Reviewing editor, and three reviewers.
There are some issues that need to be addressed before acceptance, as outlined below:
Reviewer 1 raises many important points for further clarification of the model and reviewer 2 some clear guides for improvement in terms of the justification/motivation of the models, its extrapolation to Africa and the overall messaging of the paper. The specific concerns are as follows:
Reviewer #1:
This paper combines three metaanalyses of bioassays and hut trials with analysis based on a transmission dynamic model to explore how different levels of pyrethroid resistance might be expected to affect malaria incidence and the potential benefits of switching to nets containing PBO.
Clearly this work addresses a very important public health problem. I believe the broad class of methods used are appropriate, though further justification for some specific assumptions is needed (see below). The work claims to offer substantial new insights, in particular highlighting the importance of even low levels of resistance which has the potential to change the way pyrethroid resistance is thought about. I don't know the LLIN literature well enough to assess the accuracy of this claim, but it sounds plausible.
The paper is clearly written, the results are explained very well, and the figures clearly convey the key findings. The methods are also welldescribed (though will be even clearer once code has been made available as is planned). There is one area that I think needs to be improved: while the methods describe clearly what was done, it is not always clear why. In particular, I found the reasons behind a number of modelling choices described in the subsections “Quantifying the impact of standard and PBO LLINs in the presence of insecticide resistance” and “Parameterising transmission dynamics model” opaque. For example, why use a 3rd order polynomial when looking at LLIN deterrence? Are there reasons for thinking there should be two changepoints? To what extent are the assumptions of normally distributed errors (Equation 9) justified? Can the data motivating the choice of Equations 10–11 be shown in a technical appendix? Motivation of Equations 12–14 is also lacking: taking the simplest (Equation 13) it is unclear why s_p0 (the proportion of mosquitoes feeding successfully) should depend on both m_p (the ratio of the number of mosquitoes entering a hut with a LLIN to the number entering a hut without a bednet), and on k_p (the proportion which enter and successfully feed a ptreated hut), given that the latter already includes entering. Moreover, it is claimed that s_p0 is a proportion, so should be constrained between 0 and 1, but if m_p = 1 (so bednets have no effect on entering) then s_p0=k_p/k_0, which is a ratio of two proportions rather than a proportion itself (for plausible parameters values it presumably will be below 1, but it is not necessarily so). I therefore think further motivation is required, which should ideally include some graphical assessments of model assumptions (3rd order polynomial, normally distributed errors, relationship between k_p and mortality) which could go in the appendix.
It is also good practice in any Bayesian analysis to explicitly show posterior distributions of model parameters or at the very least to summarise CrIs for these.
Reviewer #1 (Additional data files and statistical comments):
It would be useful to provide code and data used in meta analyses – and the statistical submission form states that these will be made available.
Reviewer #2:
The authors have used three small datasets (all that is available) and many assumptions to model the impact of insecticide resistance, defined using insecticide bioassay results, on malaria incidence in Africa in a defined range of scenarios. Their scope is ambitious and this work brings together several separate studies to incorporate resistance into an existing malaria model. The work builds on earlier studies and is novel and interesting, however, the results need to be caveated carefully to recognise the limitations of the datasets used and the assumptions made. It also needs to be clear that this work makes predictions for a limited range of scenarios.
The work uses a series of metaanalyses to show that insecticide bioassay results can be used to predict the impact of resistance on malaria incidence, however, the final model has not been validated using African locations that were not included in the metaanalyses datasets where values are known for both malaria incidence and resistance as measured by an insecticide bioassay.
The data used, the mosquito species included and the range of slide prevalence values considered are all specific to scenarios found in Africa but nowhere in the paper is this limitation mentioned. The implication from the title and throughout the manuscript is that this is a generalizable analysis of pyrethroid resistance but it is in fact only applicable in Africa, and only in certain scenarios within subsaharan Africa.
Throughout the Results and Discussion there are sentences that look like statements of fact but they are in fact predictions within the limits of certain scenarios defined by the authors and predicated on analysis of a limited dataset and many assumptions, for example, "For the An. gambiae complex PBO had the greatest benefit in mosquito populations with intermediate levels of pyrethroid resistance", "The probability that a mosquito will feed on someone beneath a LLIN only increases at high levels of pyrethroid resistance", and so on. These statements need to be presented as predictions, and it would also be useful if the Results could start with a summary of the scenarios covered and assumptions made (see below). In Results, the authors state "The numbers of mosquitoes deterred from entering the experimental hut substantially decreases in areas of higher pyrethroid resistance" – this is a strong statement but when you look at Figure 3A you can see that the credible intervals are large ("substantial" even) and this statement needs to caveated appropriately.
Where values are given, these are not bounded by any intervals, for example in Results "causing up to 200 additional cases per 1000" or "where over 500 cases per 100 people can be prevented each year". A range or intervals are needed to give an idea of uncertainty.
Metaanalyses 1 and 2 were compared to the observed results collected for the third metaanalysis but this comparison was visual only with no formal analysis or validation (end of subsections “Added benefit of PBO” and” Predicting the added benefit of PBO LLINs in experimental hut trials”).
The paper as a whole, and the Methods in particular, is a long and dense read and would benefit from summaries for biomedical/entomological readers who are not mathematical modellers, whilst still retaining important details about the scenarios modelled and the assumptions made. In particular, summaries aimed at these readers at the beginning of both the Methods and Results would be hugely helpful for eLife's broader readership.
Figure 1 is cited in the Introduction and seems to be a key result but is not explained in the Results at all or discussed. No credible intervals are included. It shows model predictions for the scenario where bioassay results give 20% mortality but it would be interesting to see the results for other mortality/survival rates that are often found in the region. The green dashed line in 1A shows 10% parasite prevalence but it is unclear why. The starting prevalence is >20% and then it drops after control but come up to 10% every three years so presumably this is the setting the green line refers to? The legend says the black line shows the situation with no resistance and the red line shows the situation if resistance arrives at Y6, but the red lines starts before Y0 and the black line doesn't start until Y6.
There are a lot of assumptions made by this work but it is unclear how they have been justified and which ones have been subject to sensitivity analyses in the context of the results presented here, i.e. the impact of the prevalence of resistance on malaria incidence. The assumptions made include: Assumed mosquito deterrence and exiting can be described by the degree of mosquito mortality seen in the same hut trial; Assumed the relationship between deterrence/exiting, feeding successfully and dying is consistent across all species; Assumed washing nets gives the same results as a durability study; Assumed the activity of the insecticide decays according to a given formula that includes halflife in washes; Assumed resistance arises spontaneously, and after six years of LLINs use; Assumed LLINs are redistributed every three years; Assumed transmission is perennial; Assumed there is no other vector control (and presumably no other nonvector related pyrethroid use); Assumed resistance remains constant after arising; Assumed 35% clinical cases are treated of which 36% receive ACT); Incorporated assumptions/estimates used previously by the same group that are not all given here; Assumed physiological resistance has no effect on the vectorial capacity of individual mosquitoes.
Reviewer #2 (Additional data files and statistical comments):
I am not a mathematical modeller and assume that this manuscript has also gone to reviewers who can comment on the modelling work in more depth.
The authors propose to make the data used in the metaanalysis available via Dryad if they have previously been published. This is reasonable but some of the unpublished datasets were provided by the authors of this paper and so should also be included in data deposit.
I am not qualified to comment on whether the source code provided would allow a reader to repeat this work but this is important.
Reviewer #3:
This is a well written, welldesigned and important manuscript.
The observation that bioassay survival can be used as a quantitative test to assess the level of pyrethroid resistance is an important one. As the authors note bioassays are a crude tool but they can be potential important on a programmatic level.
The observation that LLINs provide protection until high levels of resistance is important from a policy perspective and contributes to understanding a poorly studied topic.
The model is statistically sound and the authors must be commended on sharing their code for full transparency.
The final recommendation of the benefits of PBO nets has important policy implications for areas high in resistance.
https://doi.org/10.7554/eLife.16090.031Author response
There are some issues that need to be addressed before acceptance, as outlined below:
Reviewer 1 raises many important points for further clarification of the model and reviewer 2 some clear guides for improvement in terms of the justification/motivation of the models, its extrapolation to Africa and the overall messaging of the paper. The specific concerns are as follows:
Reviewer #1:
This paper combines three metaanalyses of bioassays and hut trials with analysis based on a transmission dynamic model to explore how different levels of pyrethroid resistance might be expected to affect malaria incidence and the potential benefits of switching to nets containing PBO.
Clearly this work addresses a very important public health problem. I believe the broad class of methods used are appropriate, though further justification for some specific assumptions is needed (see below). The work claims to offer substantial new insights, in particular highlighting the importance of even low levels of resistance which has the potential to change the way pyrethroid resistance is thought about. I don't know the LLIN literature well enough to assess the accuracy of this claim, but it sounds plausible.
The paper is clearly written, the results are explained very well, and the figures clearly convey the key findings. The methods are also welldescribed (though will be even clearer once code has been made available as is planned). There is one area that I think needs to be improved: while the methods describe clearly what was done, it is not always clear why. In particular, I found the reasons behind a number of modelling choices described in the subsections “Quantifying the impact of standard and PBO LLINs in the presence of insecticide resistance” and “Parameterising transmission dynamics model” opaque. For example, why use a 3rd order polynomial when looking at LLIN deterrence? Are there reasons for thinking there should be two changepoints?
We agree that the choice of curves could be more thoroughly explained. There is no a priori reason to assume that there is a changepoint though the polynomial was chosen as it provides complete flexibility for the shape of the relationship given that there is no obvious pattern to these data. This flexibility indeed shows that some of the 95% credible interval curves do have a change point though more studies would be required to thoroughly evaluate this. We have changed the text accordingly to give further justification:
“Though there is no a priori reason to assume an inflection point in the relationship between m_p and l_p the polynomial function is chosen as it is highly flexible and would allow such a curve should it exist (which is necessary given the variability in the raw data).”
To what extent are the assumptions of normally distributed errors (Equation 9) justified?
The assumption of normally distributed errors was investigated using a quantilequantile plot which indicates that these data is adequately described using a normal distribution. This figure has been included in Figure 3—figure supplement 1A whilst caption says:
“(A) shows a normal quantilequantile plot for the residuals of the data for the relationship between deterrence and mosquito survival in experimental hut trials (Figure 3A, equation). The linearity of the residuals (the proximity of the blue dots to the red dotted line) indicate that the error in these data are adequately described by the normal distribution (see Equation [9])”.
Can the data motivating the choice of Equations 10–11 be shown in a technical appendix?
The data motivating the choice of Equation [11] is shown in Figure 3C. We have made this clear in the manuscript by including the following text:
“Visual inspection of these data indicates that k_p increases with decreasing mortality at an exponential rate (Figure 3C).”
Motivation of Equations 12–14 is also lacking: taking the simplest (Equation 13) it is unclear why s_p0 (the proportion of mosquitoes feeding successfully) should depend on both m_p (the ratio of the number of mosquitoes entering a hut with a LLIN to the number entering a hut without a bednet), and on k_p (the proportion which enter and successfully feed a ptreated hut), given that the latter already includes entering.
We thank the reviewer for highlighting this section as a mistake was made when transcribing the parameter definitions from the original Griffin et al. model. k_p is actually the probability of successful feeding conditional on the mosquito having entered, not the proportion which enter and successfully feed as we initially stated. We have corrected the definition in the main text thus:
“The proportion of mosquitoes entering the hut which exit without feeding is denoted j_p whilst the proportion which successfully feed upon entering is k_p.”
Moreover, it is claimed that s_p0 is a proportion, so should be constrained between 0 and 1, but if m_p = 1 (so bednets have no effect on entering) then s_p0=k_p/k_0, which is a ratio of two proportions rather than a proportion itself (for plausible parameters values it presumably will be below 1, but it is not necessarily so).
We agree that this has not been properly explained. We have tightened up the definition of s_p0 in the main text:
“Estimates of j_p, l_p and m_p can be used to determine the proportion of mosquitoes repeating (a combination of deterrence and exiting, r_p0), dying (d_p0) and feeding successfully (s_p0) during a single feeding attempt in a hut with a new LLIN relative to those successfully feeding in a hut without an LLIN (i.e. p=1 or 2)…”.
And given a further justification for the assumptions implicit in the equations in the following paragraph (using data from the metaanalysis):
“Though theoretically s_p0 could have values >1 for practical purposes it is constrained between zero and one as on average mosquitoes entering a hut with an LLIN are less likely to feed than mosquitoes entering a hut without a bednet (as shown by all estimates of k_p being substantially lower than k_0, see Figure 3C and Table 6).”
I therefore think further motivation is required, which should ideally include some graphical assessments of model assumptions (3rd order polynomial, normally distributed errors, relationship between k_p and mortality) which could go in the appendix.
It is also good practice in any Bayesian analysis to explicitly show posterior distributions of model parameters or at the very least to summarise CrIs for these.
We agree that further graphical assessment would allow the reader to have a better understanding of the validity of the model. Therefore we have included all the figures highlighted by the reviewer. A plot justifying the use of a normal distribution when fitting m_p is included in Figure 3—figure supplement 1A whilst posterior distributions of each of the 14 other parameters is shown in Figure 3—figure supplement 1B). The relationship between k_p and mortality (in experimental hut trials) is shown in Figure 3C whilst the relationship between s_p0 and mortality (in the bioassay) is shown in Figure 4—figure supplement 1). All figures have been referred to in the main text or figure captions.
Reviewer #1 (Additional data files and statistical comments):
It would be useful to provide code and data used in meta analyses – and the statistical submission form states that these will be made available.
The code is included in the resubmission, together with the initial dataset that will be added to Dryad.
Reviewer #2:
The authors have used three small datasets (all that is available) and many assumptions to model the impact of insecticide resistance, defined using insecticide bioassay results, on malaria incidence in Africa in a defined range of scenarios. Their scope is ambitious and this work brings together several separate studies to incorporate resistance into an existing malaria model. The work builds on earlier studies and is novel and interesting, however, the results need to be caveated carefully to recognise the limitations of the datasets used and the assumptions made. It also needs to be clear that this work makes predictions for a limited range of scenarios.
The work uses a series of metaanalyses to show that insecticide bioassay results can be used to predict the impact of resistance on malaria incidence, however, the final model has not been validated using African locations that were not included in the metaanalyses datasets where values are known for both malaria incidence and resistance as measured by an insecticide bioassay.
The reviewer raises an important point and we agree that there should be further statements for the limitation of the model given the available datasets. Specific changes are outlined below. Regarding the validation of the model with field data we would very much like to do this exercise though are unaware of any current publicly available datasets which would allow such validation. To our knowledge there are no sites where insecticide bioassays are regularly measured together with malaria metrics (either prevalence or incidence) and where the history of malaria control is known. If the reviewer knows of such a dataset we would be happy to try and test out the model. Hopefully studies will be forthcoming over the next few years which would allow for elements of the model to be validated in specific locations. We have modified the following paragraph to raise the important question of model validation:
“Currently a mosquito population is defined as being pyrethroid resistant if there is <90% bioassay mortality (World Health Organisation, 2013; Viana et al., 2016). […] Though the principles outlined here may apply to other mosquito species in different settings care should be taken when extrapolating the results beyond the areas where the data were collated.”
The data used, the mosquito species included and the range of slide prevalence values considered are all specific to scenarios found in Africa but nowhere in the paper is this limitation mentioned. The implication from the title and throughout the manuscript is that this is a generalizable analysis of pyrethroid resistance but it is in fact only applicable in Africa, and only in certain scenarios within subsaharan Africa.
We agree that the work is very African centric so have changed the title of the paper to reflect this (see below). Further clarification of the geographical and species limits of the model predictions is outlined in our response above to Reviewer #1.
Throughout the Results and Discussion there are sentences that look like statements of fact but they are in fact predictions within the limits of certain scenarios defined by the authors and predicated on analysis of a limited dataset and many assumptions, for example, "For the An. gambiae complex PBO had the greatest benefit in mosquito populations with intermediate levels of pyrethroid resistance", "The probability that a mosquito will feed on someone beneath a LLIN only increases at high levels of pyrethroid resistance", and so on. These statements need to be presented as predictions, and it would also be useful if the Results could start with a summary of the scenarios covered and assumptions made (see below).
We have gone through the Results and Discussion sections and clarified what evidence we are using with all of our statements. Broadly this breaks down into data observations and model predictions. Each change has been noted below (examples where single words were switched are not included).
Results – “Data suggests that for the An. gambiae complex PBO has the greatest benefit in mosquito populations with intermediate levels of pyrethroid resistance”;
Results – “Figure 3A indicates that the number of mosquitoes deterred from entering the experimental hut substantially decreases in areas of higher pyrethroid resistance…”;
Results – “The transmission dynamics model predicts that the higher the level of pyrethroid resistance the greater impact it will have on both the number of clinical cases (Figures 4A and 4B)”;
Results – “The impact of the addition of the synergist, PBO, on pyrethroid induced mortality appears to depend on mosquito species and the level of pyrethroid resistance”;
Discussion – “The metaanalysis of experimental hut trial data suggest that the probability that a mosquito will feed on someone beneath a LLIN only increases at high levels of pyrethroid resistance (Figure 3C)”;
Discussion – “The loss of LLIN induced mosquito mortality is likely to decrease the community impact of LLINs, increasing average mosquito age and the likelihood that people are infected whilst unprotected by a bednet.”
In Results, the authors state "The numbers of mosquitoes deterred from entering the experimental hut substantially decreases in areas of higher pyrethroid resistance" – this is a strong statement but when you look at Figure 3A you can see that the credible intervals are large ("substantial" even) and this statement needs to caveated appropriately.
We agree and have changed the text accordingly:
“Figure 3A indicates that the number of mosquitoes deterred from entering the experimental hut substantially decreases in areas of higher pyrethroid resistance (where LLIN induced mortality inside the hut is low) though the variability around the best fit line is high suggesting the precise shape of the relationship is uncertain.”
Where values are given, these are not bounded by any intervals, for example in Results "causing up to 200 additional cases per 1000" or "where over 500 cases per 100 people can be prevented each year". A range or intervals are needed to give an idea of uncertainty.
We agree that credible intervals would help interpretation and have included them were necessary together with full credible intervals estimates for Figure 4—figure supplements 13 and Figure 5—figure supplement 1–3.
“For example with as little as 30% resistance (70% mortality in discriminating dose assay) in a population with 10% slide prevalence (in 210 year olds) the model predicts that pyrethroid resistance would cause an additional 245 (95%CI 142340) cases per 1000 people per year (Figure 4A, averaged over the 3 year lifeexpectancy of the net)”.
“For example in an area with 10% endemicity and 80% resistance (20% mortality in discriminating dose assay) the model predicts that switching to PBO LLINs would avert 501 (95%CI 319621) cases per 1000 people per year (Figure 5A).”
Metaanalyses 1 and 2 were compared to the observed results collected for the third metaanalysis but this comparison was visual only with no formal analysis or validation (end of subsections “Added benefit of PBO” and” Predicting the added benefit of PBO LLINs in experimental hut trials”).
We understand that some readers might like to see some sort of test to show the goodness of fit. This analysis has been done and highlighted in the Methods and caption of Figure 2C:
“Overall the model appears to be a good predictor of these data, both visually and statistically (Analysis of Variance test shows there was no significant difference between model predictions and observed data pvalue=0.35).”
The paper as a whole, and the Methods in particular, is a long and dense read and would benefit from summaries for biomedical/entomological readers who are not mathematical modellers, whilst still retaining important details about the scenarios modelled and the assumptions made. In particular, summaries aimed at these readers at the beginning of both the Methods and Results would be hugely helpful for eLife's broader readership.
We agree that the paper is dense, though given the depth required to explain the diverse range of methods we think it is essential. As suggested we have tried to summarise the Methods section by extending the outline of the paper at the end of the Introduction (see below or the extended paragraph). As for a summary of the results we feel that this will be best done as part of the eLife digest, which will be written if the paper is accepted.
“Here we propose that information on the current malaria endemicity, mosquito species and level of pyrethroid resistance (as measured by bioassay mortality) can be used to predict the public health impact of pyrethroid resistance and choosing the most appropriate LLIN for the epidemiological setting. […] Finally (4) this model is combined with bioassay and experimental hut trial results to predict the epidemiological impact of switching from mass distribution of a standard to a PBO LLIN.”
Figure 1 is cited in the Introduction and seems to be a key result but is not explained in the Results at all or discussed. No credible intervals are included. It shows model predictions for the scenario where bioassay results give 20% mortality but it would be interesting to see the results for other mortality/survival rates that are often found in the region.
Figure 1 is intended to be used to interpret the results of the transmission dynamics model. It illustrates the scenarios and introduces the variables which are changed in the later figures. Therefore it originally wasn’t highlighted in the Results, just the Introduction and Methods. Having said this we agree with the reviewer that showing reruns of the model with different parameters might be informative. Therefore we have included two additional figures supplements to Figure 1 with two different levels of pyrethroid resistance, low (20% bioassay survival, Figure 1—figure supplement 1) and high (80% bioassay mortality, Figure 1—figure supplement 2). We now present the moderate scenario (50% bioassay mortality) in Figure 1. The new figures are in addition referred to in the main Discussion.
“Detecting an epidemiological impact of a low population prevalence of resistance may be challenging for local health systems (for example, see <20% resistance prevalence Figure 1—figure supplement 1, Figure 4) especially in an area where LLIN coverage, local climatic conditions and the use of other malaria control interventions are changing over time. These simulations also assume that resistance arrives overnight, when in reality it will spread through a mosquito population more gradually and therefore may be harder to detect.”
The green dashed line in 1A shows 10% parasite prevalence but it is unclear why. The starting prevalence is >20% and then it drops after control but come up to 10% every three years so presumably this is the setting the green line refers to?
Yes, it is. It is mentioned in the caption for the figure that “Endemicity (a variable in Figures 4 and 5) is changed by varying the slide prevalence in 210 year olds at year 6 (by changing the vector to host ratio) and in this plot takes a value of 10% (as illustrated by the horizontal green dashed line in A).” To stress the importance of Figure 1 to the interpretation of later Figures we have added the following section to the end of the Introduction:
“Thirdly, information from (1) and (2) is used to parameterise a widely used malaria transmission dynamics mathematical model to estimate public health impact of pyrethroid resistance in different settings taking into account the community impact of LLINs. An illustration of model predictions showing how different malaria metrics change over time is given in Figure 1. The figure also indicates how LLIN coverage and variables such as malaria endemicity and pyrethroid resistance are incorporated in the model.”
The legend says the black line shows the situation with no resistance and the red line shows the situation if resistance arrives at Y6, but the red lines starts before Y0 and the black line doesn't start until Y6.
In the original figure the red line (resistance population) overran the black line (susceptible population) until Y6 when they diverge. To avoid confusion we have added a little jitter so that both lines are visible throughout.
There are a lot of assumptions made by this work but it is unclear how they have been justified and which ones have been subject to sensitivity analyses in the context of the results presented here, i.e. the impact of the prevalence of resistance on malaria incidence.
We agree that there are lots of assumptions which are generally made due to lack of available data. When this is the case we have resorted to the simplest explanation first though each have been expanded upon in the text in turn.
“Visual inspection of these data indicates that mosquito deterrence and exiting can be described by the degree of mosquito mortality seen in the same hut trial.”
“As there is insufficient information to generate these functions for each species separately it is assumed the relationship between deterrence/exiting, feeding successfully and dying is consistent across all species”.
“For simplicity and following (Griffin et al.) it is assumed that the killing activity of pyrethroid over time is proportional to the loss of morbidity caused by washing…”.
“Following Griffin et al. itis assumed that the activity of the insecticide decays at a constant rate according to…”.
The assumptions made include: Assumed mosquito deterrence and exiting can be described by the degree of mosquito mortality seen in the same hut trial; Assumed the relationship between deterrence/exiting, feeding successfully and dying is consistent across all species; Assumed washing nets gives the same results as a durability study; Assumed the activity of the insecticide decays according to a given formula that includes halflife in washes; Assumed resistance arises spontaneously, and after six years of LLINs use; Assumed LLINs are redistributed every three years; Assumed transmission is perennial; Assumed there is no other vector control (and presumably no other nonvector related pyrethroid use); Assumed resistance remains constant after arising; Assumed 35% clinical cases are treated of which 36% receive ACT);
Each of the above assumptions are made to produce a generalisable site in Africa as in reality every geographical location will have different seasonality, level of treatment and vector control interventions and pyrethroid resistance profile. It is infeasible to do a sensitivity analysis on all these assumptions and unlikely to provide the reader with an additional insight. Therefore throughout the manuscript we only vary the 3 most informative settings, mosquito species, malaria endemicity and level of pyrethroid resistance (as measured by a bioassay).
Incorporated assumptions/estimates used previously by the same group that are not all given here;
The number of individual assumptions and parameter estimates made in the Griffin et al. and Walker et al. papers are considerable and including them all here would make this manuscript unwieldly. Therefore we refer the reader to the original references.
Assumed physiological resistance has no effect on the vectorial capacity of individual mosquitoes.
There are a multitude of ways in which resistance might impact vectorial capacity other than changing the susceptibility of mosquitoes to insecticide. However, to our knowledge none of these have been shown to occur in wild caught mosquitoes so including them at this stage would be premature and is beyond the scope of this paper. We have stressed this in the following sentence at the start of the Results:
“It is assumed that the ability of a mosquito to survive insecticide exposure is not associated with any other behavioural or physiological change in the mosquito population which influences malaria transmission. For example, an increased propensity for mosquitoes to feed outdoors (subsequently referred to as behavioural resistance) would limit their exposure to LLINs though there is currently insufficient field evidence to justify its inclusion in the model (Briet and Chitnis, 2013; Gatton et al., 2013).”
Reviewer #2 (Additional data files and statistical comments):
I am not a mathematical modeller and assume that this manuscript has also gone to reviewers who can comment on the modelling work in more depth.
The authors propose to make the data used in the metaanalysis available via Dryad if they have previously been published. This is reasonable but some of the unpublished datasets were provided by the authors of this paper and so should also be included in data deposit.
We agree that ideally all data would be made immediately publicly available on acceptance of this manuscript. All bioassay data (published and unpublished) shall be uploaded to Dryad though some experimental hut trial data are currently unavailable for inclusion. We contacted all custodians of the unpublished experimental hut trial data and a number of them would prefer for their raw data not to be made publicly available as they are currently in the process of publishing it themselves. Once these manuscripts have been published we will update the Dryad dataset so that it includes all data points reviewed in this study.
https://doi.org/10.7554/eLife.16090.032Article and author information
Author details
Funding
Medical Research Council
 Thomas S Churcher
Department for International Development
 Thomas S Churcher
Innovative Vector Control Consortium
 Thomas S Churcher
Wellcome Trust (ISSF Grant)
 Natalie Lissenden
European Research Council (265660)
 Hilary Ranson
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Acknowledgements
TSC would like to thank the IVCC (Innovative Vector Control Consortium) and the UK Medical Research Council (MRC) / UK Department for International Development (DFID) under the MRC/DFID Concordat agreement. The financial support of the European Union Seventh Framework Programme FP7 (2007–2013) under grant agreement no 265660 AvecNet is gratefully acknowledged. NL was supported by an ISSF Grant from the Wellcome Trust.
Reviewing Editor
 Simon I Hay, Institute for Health Metrics and Evaluation, United States
Publication history
 Received: March 16, 2016
 Accepted: August 18, 2016
 Accepted Manuscript published: August 22, 2016 (version 1)
 Version of Record published: September 15, 2016 (version 2)
Copyright
© 2016, Churcher et al.
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics

 4,535
 Page views

 849
 Downloads

 137
 Citations
Article citation count generated by polling the highest count across the following sources: Scopus, Crossref, PubMed Central.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading

Modelling the effectiveness of bednets against mosquitoes and malaria.

 Epidemiology and Global Health
Background:
Shortterm forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID19 cases and deaths across Europe between 08 March 2021 and 07 March 2022.
Methods:
We used opensource tools to develop a public European COVID19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID19 cases and deaths reported by a standardised source for 32 countries over the next 1–4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equallyweighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance.
Results:
Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models’ forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models’ forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models’ forecasts of deaths (N=763 predictions from 20 models). Across a 1–4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models.
Conclusions:
Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks.
Funding:
AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS1633028, NSF Grant No.: OAC1916805, NSF Expeditions in Computing Grant CCF1918656, CCF1917819, NSF RAPID CNS2028004, NSF RAPID OAC2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA119D0007, and respectively Virginia Dept of Health Grant VDH215010141, VDH215010143, VDH215010147, VDH215010145, VDH215010146, VDH215010142, VDH215010148. AF, AMa, GL funded by SMIGE  Modelli statistici inferenziali per governare l'epidemia, FISR 2020Covid19 I Fase, FISR2020IP00156, Codice Progetto: PRJ0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission  DirectorateGeneral for Communications Networks, Content and Technology through the contract LC01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018095456BI00. DE., MGu funded by Spanish Ministry of Health / REACTUE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID19 (https://www.nfdi4health.de/taskforcecovid192) within the framework of a DFGproject (LO342/171). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for realtime monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).