Replication Study: The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors

Article
Figures and data
Abstract
Introduction
Results and discussion
Materials and methods
References
Decision letter
Author response
Article and author information
Metrics

Abstract

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Chroscinski et al., 2015) that described how we intended to replicate selected experiments from the paper “The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors “(Willingham et al., 2012). Here we report the results of those experiments. We found that treatment of immune competent mice bearing orthotopic breast tumors with anti-mouse CD47 antibodies resulted in short-term anemia compared to controls, consistent with the previously described function of CD47 in normal phagocytosis of aging red blood cells and results reported in the original study (Table S4; Willingham et al., 2012). The weight of tumors after 30 days administration of anti-CD47 antibodies or IgG isotype control were not found to be statistically different, whereas the original study reported inhibition of tumor growth with anti-CD47 treatment (Figure 6A,B; Willingham et al., 2012). However, our efforts to replicate this experiment were confounded because spontaneous regression of tumors occurred in several of the mice. Additionally, the excised tumors were scored for inflammatory cell infiltrates. We found IgG and anti-CD47 treated tumors resulted in minimal to moderate lymphocytic infiltrate, while the original study observed sparse lymphocytic infiltrate in IgG-treated tumors and increased inflammatory cell infiltrates in anti-CD47 treated tumors (Figure 6C; Willingham et al., 2012). Furthermore, we observed neutrophilic infiltration was slightly increased in anti-CD47 treated tumors compared to IgG control. Finally, we report a meta-analysis of the result.

https://doi.org/10.7554/eLife.18173.001

Introduction

The Reproducibility Project: Cancer Biology (RP:CB) is a collaboration between the Center for Open Science and Science Exchange that seeks to address concerns about reproducibility in scientific research by conducting replications of selected experiments from a number of high-profile papers in the field of cancer biology (Errington et al., 2014). For each of these papers a Registered Report detailing the proposed experimental designs and protocols for the replications was peer reviewed and published prior to data collection. The present paper is a Replication Study that reports the results of the replication experiments detailed in the Registered Report (Chroscinski et al., 2015) for a paper by Willingham et al., and uses a number of approaches to compare the outcomes of the original experiments and the replications.

In 2012, Willingham et al. reported that blocking the signal regulatory protein alpha (SIRPa)/CD47 interaction with an anti-CD47 blocking antibody promoted phagocytosis of solid tumor cells in vitro and reduced growth of solid tumors in vivo indicating that anti-CD47 antibody therapy may be an effective treatment for a variety of solid tumors. Using a syngeneic breast cancer model, mouse anti-CD47 antibody treatment resulted in a statistically significant decrease in final tumor weight compared to IgG isotype control (Willingham et al., 2012). Anti-CD47 treatment also increased lymphocytic infiltration to the tumor site without unacceptable toxicity except short-term anemia observed immediately after dosing.

The Registered Report for the paper by Willingham et al. described the experiments to be replicated (Figure 6A–C and Table S4), and summarized the current evidence for these findings (Chroscinski et al., 2015). Since that publication there have been additional studies examining the safety and efficacy of targeting CD47 as an anti-cancer therapeutic. Anti-CD47 treatment was reported to increase macrophage phagocytosis, decrease tumor weight, and inhibit spontaneous metastasis in a osteosarcoma xenograft model (Xu et al., 2015). Similarly, CD47 blockade was reported to enhance tumor cell phagocytosis by macrophages, reduce tumor burden, and increase survival in glioblastoma (Zhang et al., 2016), gastric cancer (Yoshida et al., 2015), and pancreatic neuroendocrine tumor (Krampitz et al., 2016) xenograft models. Cioffi and colleagues tested the effect of inhibiting CD47 in pancreatic ductal adenocarcinoma (PDAC) and reported that while anti-CD47 antibodies increased phagocytosis in vitro, it did not result in a statistically significant change in tumor growth in a PDAC patient-derived xenograft (PDX) model unless administered in combination with a chemotherapeutic agent (Cioffi et al., 2015). Additionally, a humanized anti-CD47 antibody was tested for safety and efficacy in disease models of acute myeloid leukemia (AML) and was reported to decrease tumor burden and increase survival in an AML PDX model (Liu et al., 2015). A pre-clinical toxicokinetic study in non-human primates reported no adverse effects associated with the humanized antibody (Liu et al., 2015) and patients with AML and solid tumors are being recruited for phase one clinical trials (ClinicalTrials.gov identifiers: NCT02678338 and NCT02216409).

The outcome measures reported in this Replication Study will be aggregated with those from the other Replication Studies to create a dataset that will be examined to provide evidence about reproducibility of cancer biology research, and to identify factors that influence reproducibility more generally.

Results and discussion

Engraftment of mouse breast cancer cells and treatment with CD47 targeting antibodies

We sought to independently replicate the safety and efficacy of targeting CD47 in immune competent mice using a syngeneic model of breast cancer. This experiment is similar to what was reported in Figure 6A–C of Willingham et al. (2012). MT1A2 mouse breast cancer cells (Addison et al., 1995) were engrafted into the mammary fat pad of syngeneic FVB mice and monitored until palpable tumors formed. Mice with palpable tumors were randomized to receive injections of either 400 µg mouse IgG isotype control (IgG) or 400 µg anti-mouse CD47 (anti-CD47) antibodies every other day into the mammary fat pad proximal to the tumor. While the original study included two clones of anti-CD47 (MIAP410 [Han et al., 2000] and MIAP301 [Lindberg et al., 1993]), this replication attempt was restricted to only one clone, MIAP410, which had the larger reported effect size of the two clones in the original experiment (Willingham et al., 2012).

To test any potential toxicity of the antibody treatment, hematological analysis was performed on blood collected by retro-orbital bleeding 5 days after the beginning of antibody injections. Untreated FVB mice were used to determine the baseline reading. This differed from the original study, which analyzed toxicity in different mouse strains and treatment regimens. In the original study, BALB/c mice were analyzed for blood toxicity 5 days after two 500 µg antibody injections (Table S4) and C57BL/6 mice were analyzed for specific hematology parameters 1, 3, and 6 days after a single intraperitoneal injection of 250 µg of IgG or anti-CD47 antibodies (Supplemental Figure 6) (Willingham et al., 2012). Similar to what was reported for C57BL/6 mice in the original paper, FVB mice treated with anti-CD47 resulted in short-term anemia (Figure 1; Figure 1—figure supplement 1). Red blood cell count (Figure 1G; one-way ANOVA; F(2,17) = 424.9, uncorrected p=3.07×10⁻¹⁵ with alpha level = 0.0033; (Bonferroni corrected p=4.60×10⁻¹⁴)), hemoglobin (Figure 1H; one-way ANOVA; F(2,17) = 502.1, uncorrected p=7.61×10⁻¹⁶ with alpha level = 0.0033; (Bonferroni corrected p=1.14×10⁻¹⁴)), and hematocrit (Figure 1I; one-way ANOVA; F(2,17) = 283.0, uncorrected p=8.93×10⁻¹⁴ with alpha level = 0.0033; (Bonferroni corrected p=1.34×10⁻¹²)) were all slightly reduced in anti-CD47 treated mice compared to untreated and IgG treated mice. This is consistent with the previously described function of CD47 in the normal phagocytosis of aging red blood cells (Oldenborg et al., 2000; Oldenborg et al., 2001; Oldenborg, 2004) and has been observed in other studies examining anti-CD47 antibody treatment (Liu et al., 2015). Additionally, three animals treated with anti-CD47 showed mild monocytosis (Figure 1D; Figure 1—figure supplement 1).

Figure 1 with 1 supplement see all

Download asset Open asset

Blood toxicity analysis.

Female FVB mice bearing orthotopic MT1A2 breast tumors were randomized to receive IgG isotype control (IgG) (n = 7) or anti-mouse CD47 (CD47) (n = 7) antibodies. Mice bearing small or undetectable tumors were designated for baseline reading (NT) (n = 6). Five days after the beginning of treatment blood samples collected via retro-orbital bleed were analyzed on a hematology analyzer. Dot plots with means reported as crossbars for each hematological parameter. For each parameter a one-way ANOVA was performed and the alpha level or p-value was adjusted using the Bonferroni correction. (A) White blood cells (WBC), one-way ANOVA; F(2,17) = 15.09, uncorrected p=0.00017 with alpha level = 0.0033; (Bonferroni corrected p=0.0026). (B) Neutrophils (NE), one-way ANOVA; F(2,17) = 1.02, uncorrected p=0.381 with alpha level = 0.0033; (Bonferroni corrected p>0.99). (C) Lymphocytes (LY), one-way ANOVA; F(2,17) = 20.84, uncorrected p=2.67×10⁻⁵ with alpha level = 0.0033; (Bonferroni corrected p=0.00040). (D) Monocytes (MO), Welch's one-way ANOVA; F(2,8.52) = 9.98, uncorrected p=0.0058 with alpha level = 0.0033; (Bonferroni corrected p=0.0877). (E) Eosinophils (EO), one-way ANOVA; F(2,17) = 0.06, uncorrected p=0.942 with alpha level = 0.0033; (Bonferroni corrected p>0.99). (F) Basophils (BA), one-way ANOVA; F(2,17) = 4.20, uncorrected p=0.0330 with alpha level = 0.0033; (Bonferroni corrected p=0.495). (G) Red blood cells (RBC), one-way ANOVA; F(2,17) = 424.9, uncorrected p=3.07×10⁻¹⁵ with alpha level = 0.0033; (Bonferroni corrected p=4.60×10⁻¹⁴). (H) Hemoglobin (Hb), one-way ANOVA; F(2,17) = 502.1, uncorrected p=7.61×10⁻¹⁶ with alpha level = 0.0033; (Bonferroni corrected p=1.14×10⁻¹⁴). (I) Hematocrit (HCT), one-way ANOVA; F(2,17) = 283.0, uncorrected p=8.93×10⁻¹⁴ with alpha level = 0.0033; (Bonferroni corrected p=1.34×10⁻¹²). (J) Mean corpuscular volume (MCV), one-way ANOVA; F(2,17) = 11.81, uncorrected p=0.00061 with alpha level = 0.0033; (Bonferroni corrected p=0.0091). (K) Mean corpuscular hemoglobin (MCH), one-way ANOVA; F(2,17) = 10.64, uncorrected p=0.00101 with alpha level = 0.0033; (Bonferroni corrected p=0.0151). (L) Mean corpuscular hemoglobin concentration (MCHC), one-way ANOVA; F(2,17) = 0.61, uncorrected p=0.552 with alpha level = 0.0033; (Bonferroni corrected p>0.99). (M) Red blood cell distribution width (RDW), Welch's one-way ANOVA; F(2,10.46) = 30.62, uncorrected p=4.25×10⁻⁵ with alpha level = 0.0033; (Bonferroni corrected p=0.00064). (N) Platelets (PLT), one-way ANOVA; F(2,17) = 0.62, uncorrected p=0.548 with alpha level = 0.0033; (Bonferroni corrected p>0.99). (O) Mean platelet volume (MPV), one-way ANOVA; F(2,17) = 6.98, uncorrected p=0.0061 with alpha level = 0.0033; (Bonferroni corrected p=0.092). Additional details for this experiment can be found at https://osf.io/g57ch/.

https://doi.org/10.7554/eLife.18173.002

After 30 days of antibody treatment, tumors were excised and weighed (Figure 2). Tumors treated with IgG grew to an average of 0.075 grams [n = 7, SD = 0.078], while tumors treated with anti-CD47 resulted in an average weight of 0.163 grams [n = 6, SD = 0.096]. The comparison of these two groups was not statistically significant (Welch's t-test; t(9.66) = 1.796, p=0.104). This is in comparison to the original study, which reported an average weight of 0.144 grams [n = 5, SD = 0.052] for IgG treated tumors and an average of 0.012 grams [n = 5, SD = 0.002] in anti-CD47 treated tumors. The range of observed tumor weights in the original study varied from 9 to 198 mg, with IgG treated tumors representing the higher observed weights (60–198 mg) while anti-CD47 treated tumors were reported between 9 and 14 mg. This compares to this replication attempt which observed tumor weights ranging from 5 to 257 mg, with IgG treated tumors (5–203 mg) and anti-CD47 treated tumors (41–257 mg) having fairly similar wide distributions. Indeed, the relative standard deviation (RSD) associated with this replication attempt (IgG treated =104%; anti-CD47 treated =59%) was larger than the RSD reported in the original study (IgG treated =36%; anti-CD47 treated =18%). The RSD of the IgG treated tumors reported in Willingham et al. (2012) is similar to the estimated RSDs (~30%) in the control conditions from two other published studies that utilized MT1A2 cells (Ahn and Brown, 2008; Noblitt et al., 2005), granted these studies injected more cells and in different sites than the original study and this replication attempt. Interestingly, a more recent paper briefly stated that they purposefully did not utilize the MT1A2 cell line in their study because of a high prevalence of spontaneous tumor regression, confounding the results (Desilva et al., 2012). An evaluation of tumor growth revealed this also occurred in this replication attempt with three tumors regressing at the end of the study compared to the last tumor volume measurement taken 14 days after the start of treatment (Figure 2—figure supplement 1). While these observations confound the results of this replication attempt, we further explored the tumor weight data by conducting the same analysis above, but with the three tumors that regressed during the course of the study removed. This was also not statistically significant (Welch's t-test; t(7.94) = 0.745, p=0.478, Glass' $Δ$ = −0.58, 95% CI [−1.88, 0.80]).

Figure 2 with 1 supplement see all

Download asset Open asset

Final tumor weights of immune competent hosts treated with control or CD47 targeted antibodies.

At the end of the predefined study period (Day 31), tumors from mice bearing orthotopic MT1A2 breast tumors treated every other day with IgG isotype control (IgG) (n = 7) or anti-mouse CD47 (anti-CD47) (n = 6) antibodies were excised and weighed. Dot plot with means reported as crossbars and error bars represent s.e.m. Two-tailed Welch’s t-test between IgG and anti-CD47 treated tumors; t(9.66) = 1.796, p=0.104. Additional details for this experiment can be found at https://osf.io/g57ch/.

https://doi.org/10.7554/eLife.18173.004

Dissected tumors were further processed and H&E-stained histological sections were blindly analyzed for the extent of lymphocytic infiltration. This is similar to the original study, however this replication attempt used a predefined scoring system to assess the degree of lymphocytic infiltrate (Demaria et al., 2001). Both IgG and anti-CD47 treated tumors resulted in minimal to moderate lymphocytic infiltrate (Table 1). Although not planned, the tumors were also analyzed for the extent of neutrophilic infiltration. The neutrophilic infiltrate in IgG treated tumors was minimal in 4 and moderate in 3 tumors. The neutrophilic infiltrate in anti-CD47 treated tumors were minimal in 1, moderate in 3, and brisk in 2 tumors (Table 1).

Table 1

Severity of inflammatory cell infiltration of tumors. Excised tumors were fixed, sectioned, and stained with hematoxylin and eosin and blindly scored by a Board Certified pathologist utilizing the severity score for inflammatory cell infiltrates (Demaria et al., 2001). Tumor infiltrating lymphocytes and neutrophils were scored for tumors from mice bearing orthotopic MT1A2 breast tumors treated every other day with IgG isotype control (IgG) (n = 7) or anti-mouse CD47 (CD47) (n = 6) antibodies. Additional details for this experiment can be found at https://osf.io/g57ch/.

https://doi.org/10.7554/eLife.18173.006

	Lymphocytic infiltrate				Neutrophilic infiltrate
Treatment	Absent	Minimal	Moderate	Brisk	Absent	Minimal	Moderate	Brisk
IgG	0	6	1	0	0	4	3	0
CD47	0	5	1	0	0	1	3	2

There are a number of factors that can affect tumor growth. While tumor growth is exponential under an ideal scenario, factors such as availability of nutrients, oxygen, and space influence and alter the growth of the tumor initially compared to the continued growth of the tumor (Cornelis et al., 2013; Talkington and Durrett, 2015). Simultaneously, other murine immunogenic tumor models are known to spontaneously regress (Penichet et al., 2001; Robinson et al., 2009; Vince et al., 2004), which is a phenomenon known to naturally occur in cancer patients (Jessy, 2011; Saleh et al., 2005; Salman, 2016) .

Meta-analysis of original and replicated effects

We performed a meta-analysis using a random-effects model for the effect described above as pre-specified in the confirmatory analysis plan (Chroscinski et al., 2015). To provide a standardized measure of the effect, a common effect size Glass' $Δ$ was calculated for the original and replication study. Glass' $Δ$ is the standardized difference between two means using the standard deviation of only the control group. It is used in this case because of the unequal variance between the control and treatment conditions in the original study.

The comparison of IgG treated tumors compared to anti-CD47 treated tumors resulted in Glass' $Δ$ = 2.54, 95% CI [0.40, 4.60] for the data reported in Figure 6B of the original study (Willingham et al., 2012). This compares to Glass' $Δ$ = −1.13, 95% CI [−2.35, 0.16] reported in this study. A meta-analysis (Figure 3) of these two effects resulted in Glass' $Δ$ = 0.60, 95% CI [−3.00, 4.19], p=0.745. The effects for each study are in opposite directions and the point estimate of the replication effect size is not within the confidence interval of the original result, or vice versa. The random effects meta-analysis did not result in a statistically significant effect. Further, the Cochran's Q test for heterogeneity was statistically significant (p=0.0039), which along with a large confidence interval around the weighted average effect size from the meta-analysis suggests heterogeneity between the original and replication studies.

Figure 3

Download asset Open asset

Meta-analysis of effect.

Effect size (Glass' $Δ$ ) and 95% confidence interval are presented for Willingham et al. (2012), this replication attempt (RP:CB), and a meta-analysis to combine the two effects of tumor weight comparisons. Sample sizes used in Willingham et al. (2012) and this replication attempt are reported under the study name. Random effects meta-analysis of tumors treated with IgG compared to anti-CD47 (meta-analysis p=0.745). Additional details for this meta-analysis can be found at https://osf.io/ha2bx/.

https://doi.org/10.7554/eLife.18173.007

This direct replication provides an opportunity to understand the present evidence of these effects. Any known differences, including reagents and protocol differences, were identified prior to conducting the experimental work and described in the Registered Report (Chroscinski et al., 2015). However, this is limited to what was obtainable from the original paper and through communication with the original authors, which means there might be particular features of the original experimental protocol that could be critical, but unidentified. So while some aspects, such as cell line, strain and sex of mice, number of cells injected, and the injection site of antibody treatment were maintained, others were unknown or not easily controlled for. These include variables such as cell line genetic drift (Hughes et al., 2007; Kleensang et al., 2016), circadian biological responses to therapy (Fu and Kettner, 2013), mouse strain stocks (Clayton and Collins, 2014), housing temperature in mouse facilities (Kokolus et al., 2013), and the obesity and microbiome of recipient mice (Klevorn and Teague, 2016; Macpherson and McCoy, 2015). Additionally, a differential response to immunotherapy can occur due to heterogeneity in individual tumor microenvironments (Grosso and Jure-Kunkel, 2013), which has also been observed in the clinical setting (Ascierto and Marincola, 2014; Stevenson, 2014). Whether these or other factors influence the outcomes of this study is open to hypothesizing and further investigation, which is facilitated by direct replications and transparent reporting.

Materials and methods

As described in the Registered Report (Chroscinski et al., 2015), we attempted a replication of the experiments reported in Figure 6A–C and Supplemental Table S4 of Willingham et al. (2012). A detailed description of all protocols can be found in the Registered Report (Chroscinski et al., 2015). Additional detailed experimental notes, data, and analysis are available on the Open Science Framework (OSF) (RRID: SCR_003238) (https://osf.io/9pbos/; Horrigan et al., 2016).

Cell culture

Request a detailed protocol

MT1A2 cells (shared by Weissman lab, Stanford University) were maintained in Dulbecco's Modified Eagle's Medium (DMEM) supplemented with 4 mM L-glutamine, 10% Fetal Bovine Serum (FBS) (Sigma-Aldrich, cat # F0392), 100 U/ml penicillin, and 100 µg/ml streptomycin. All cells were grown at 37°C in a humidified atmosphere at 5% CO₂.

Quality control data for the MT1A2 cell line are available on the OSF (https://osf.io/9r5hy/). This includes results confirming the cell line was free of mycoplasma contamination and common mouse pathogens. Additionally, STR DNA profiling of the cell line was performed.

Therapeutic antibodies

Request a detailed protocol

Mouse anti-CD47, MIAP410 antibody (Weissman lab, Stanford University) and mouse IgG endotoxin depleted Protein A purified protein (Innovative Research, cat # IR-MS-GF-ED, lot # 111214DG, RRID: AB_1501657) were diluted with ice-cold PBS to a final concentration of 4.0 mg/ml and aliquoted into 18 vials of 0.8 ml each and stored at 4°C. One vial was used for each injection day and any remainder discarded. An enzyme-linked immunosorbent assay (ELISA) was performed with the mouse anti-CD47, MIAP410 antibody on plates coated with human CD47-Fc or mouse CD47-Fc (shared by Weissman lab, Stanford University). ELISA protocol details and data are available at https://osf.io/werk5/.

Animals

All animal procedures were approved by the Noble Life Sciences IACUC# 15-05-001SCI and were in accordance with Noble Life Sciences policies on the care, welfare, and treatment of laboratory animals, which adhere to the regulations outlined in the USDA Animal Welfare Act (9 CFR Parts 1,2, and 3) and the conditions specified in the Guide for the Care and Use of Laboratory Animals (National Research Council (US) Committee for the Update of the Guide for the Care and Use of Laboratory Animals, 2011).

Mice were offered Certified Rodent Diet (Harlan Teklad, cat # 2018) ad libitum. The animal room was set to maintain between 19–22°C, a relative humidity of 40–65%, and a 12 hr light/dark cycle, which was interrupted for study-related activities.

A total of 20, six-eight week old female FVB mice (Charles River, strain code 207) were inoculated orthotopically with 50,000 MT1A2 cells at a density of 5×10⁴ cells in 100 µl of FACS buffer with 25% vol/vol high concentration Matrigel (BD/Corning, cat # 354262, lot # 4090005) in the left abdominal mammary #4 fat pad. Mice were monitored every other day for signs of tumor growth. Once tumor growth was detected in any animal, tumors were measured using a digital caliper and body weights recorded. At the time of randomization, 17 animals had palpable tumors, with the 14 animals having the largest tumors entered into the study. Mice were ranked according to tumor volume from highest to lowest and the 14 animals with the largest tumors were assigned to a group using an alternating serpentine method. Designation of IgG or anti-CD47 to group 1 or 2 was done by randomly assigning the two treatment groups into one block using www.randomization.com (seed number = 21473) with variability of tumor volume measurements evenly distributed among the two conditions (Student’s t-test; t(12) = 0.44, p=0.67). The remaining six mice not randomized to treatment were used to generate baseline readings for hematological analysis with one of these animals developing a tumor before the blood analysis was performed, which means of the 20 mice inoculated, 18 developed tumors. An initial attempt with 14 mice inoculated with MT1A2 cells was terminated because too few animals (10 out of 14 animals) had established tumors. Further details can be found in the ‘Deviations from Registered Report’ section and at https://osf.io/zch4n/.

Starting on the day of randomization and every other day for 30 days, 400 µg of anti-CD47 or IgG in a 100 µl volume was injected into the mammary fat pad approximately 2 mm proximal to tumor with a 30-gauge needle and 0.5 ml syringe. One animal receiving mouse anti-CD47 antibody was found dead, with no visible cause of death, before the end of the study. No body weight loss or behavioral changes were noted in any animals. At the end of the treatment period, on study day 31, animals were anesthetized with isoflurane, sacrificed, and tumors were excised, cleaned of surrounding fat tissue, and weighed. Tumor volume was calculated from caliper measurements using the formula (volume = 1/2(length*width²) or calculated from weight of excised tumors and density (1.05 g/ml) (Jensen et al., 2008).

Hematological analysis

Request a detailed protocol

Five days after the beginning of injections mice were anesthetized with isoflurane and 0.1–0.2 ml of blood was collected from the retro-orbital sinus using a microcapillary pipet into a collection tube containing kEDTA. Samples were rocked gently and analyzed within 2 hr using a Hemavet 950FS (Drew Scientific, Miami Lakes, Florida) using the built-in mouse program to determine complete blood count parameters (15 total parameters reported). Hematology profiles are available at https://osf.io/ucxwj/.

Histopathology

Request a detailed protocol

Excised tumors were formalin-fixed, paraffin blocked, sectioned (two 5-micron thick sections of each tumor), and stained with hematoxylin and eosin as described in the Registered Report (Chroscinski et al., 2015). Sections were blindly examined microscopically by Alexander DePaoli, DVM, PhD (IDEXX Laboratories, Inc.) utilizing the severity score for inflammatory cell infiltrates (Demaria et al., 2001) where: absent = 0, minimal = 1, moderate = 2, brisk = 3. Histopathology report is available at https://osf.io/xky96/. H&E stained tumor sections are available at https://osf.io/43jau/. Remaining tumor sections are available upon request.

Statistical analysis

Request a detailed protocol

Statistical analysis was performed with R software (RRID: SCR_001905), version 3.3.1 ( R Core Team, 2016). All data, csv files, and analysis scripts are available on the OSF (https://osf.io/9pbos/). Confirmatory statistical analysis was pre-registered (https://osf.io/9gykv/) before the experimental work began as outlined in the Registered Report (Chroscinski et al., 2015). Data were checked to ensure assumptions of statistical tests were met. The hematological parameters were analyzed by multiple one-way ANOVAs; one for each of the 15 parameters analyzed. The Bonferroni correction, to account for multiple testings, was applied to the alpha error or the p-value. The Bonferroni corrected value was determined by divided the uncorrected value by the number of tests performed. For the alpha error this resulted in. 0033 (.05/15). A meta-analysis of a common original and replication effect size was performed with a random effects model and the metafor R package (Viechtbauer, 2010) (available at https://osf.io/ha2bx/). The original study data was shared by the original authors a priori during preparation of the experimental design. The data was published in the Registered Report (Chroscinski et al., 2015) and was used in the power calculations to determine the sample size for this study.

Deviations from registered report

Request a detailed protocol

The type of high concentration Matrigel was different than what is listed in the Registered Report. The Registered Report listed the High Concentration Matrigel (BD/Corning, cat # 354248) while the replication experiment used the High Concentration, Phenol Red-Free Matrigel (BD/Corning, cat # 354262). The type of Matrigel used in the original experiment was not specified. The mouse IgG protein A purified protein used in this replication experiment was endotoxin depleted while the Registered Report did not indicate this purification methodology. This was clarified during communication with the original authors prior to performing the experiment. Additional materials and instrumentation not listed in the Registered Report, but needed during experimentation are also listed.

An initial attempt to inoculate 14 animals with MT1A2 cells as outlined in the Registered Report resulted in only 10 animals with established tumors. This was terminated because the predefined number of animals (7 per group) with established tumors was not reached. For the second attempt, which is reported here, the number of animals to inoculate with MT1A2 cells was increased to 20, based on the observed rate of engraftment in the first attempt. This attempt resulted in 17 animals with detectable tumors at the time of randomization, with the 14 animals having the largest tumors assigned to IgG or anti-CD47 treatment. The remaining 6 animals were used to generate baseline readings for hematological analysis with one of these animals developing a tumor before the blood analysis was performed, which means that 18 of the 20 mice inoculated developed tumors.

The Registered Report described 13 hematological parameters, similar to what was reported in Willingham et al. (2012) (Table S4), while this replication attempt reported 15 parameters. The difference stems from an inclusive measure of granulocytes used in the original study, while this replication attempt measured the number of the three principal types of granulocytes: neutrophils, basophils, and eosinophils.

References

1. Addison CL
2. Braciak T
3. Ralston R
4. Muller WJ
5. Gauldie J
6. Graham FL
(1995) Intratumoral injection of an adenovirus expressing interleukin 2 induces regression and immunity in a murine breast cancer model
PNAS 92:8522–8526.
https://doi.org/10.1073/pnas.92.18.8522
- PubMed
- Google Scholar
1. Ahn GO
2. Brown JM
(2008) Matrix metalloproteinase-9 is required for tumor vasculogenesis but not for angiogenesis: role of bone marrow-derived myelomonocytic cells
Cancer Cell 13:193–205.
https://doi.org/10.1016/j.ccr.2007.11.032
- PubMed
- Google Scholar
1. Ascierto PA
2. Marincola FM
(2014) What have we learned from cancer immunotherapy in the last 3 years?
Journal of Translational Medicine 12:141.
https://doi.org/10.1186/1479-5876-12-141
- PubMed
- Google Scholar
(2015) Registered report: The CD47-signal regulated protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors
eLife 4:e04586.
https://doi.org/10.7554/eLife.04586
- Google Scholar
1. Cioffi M
2. Trabulo S
3. Hidalgo M
4. Costello E
5. Greenhalf W
6. Erkan M
7. Kleeff J
8. Sainz B
9. Heeschen C
(2015) Inhibition of CD47 effectively targets pancreatic Cancer Stem Cells via dual Mechanisms
Clinical Cancer Research 21:2325–2337.
https://doi.org/10.1158/1078-0432.CCR-14-1399
- PubMed
- Google Scholar
1. Clayton JA
2. Collins FS
(2014) Policy: NIH to balance sex in cell and animal studies
Nature 509:282–283.
https://doi.org/10.1038/509282a
- PubMed
- Google Scholar
1. Cornelis F
2. Saut O
3. Cumsille P
4. Lombardi D
5. Iollo A
6. Palussiere J
7. Colin T
(2013) In vivo mathematical modeling of tumor growth from imaging data: soon to come in the future?
Diagnostic and Interventional Imaging 94:593–600.
https://doi.org/10.1016/j.diii.2013.03.001
- PubMed
- Google Scholar
1. Demaria S
2. Volm MD
3. Shapiro RL
4. Yee HT
5. Oratz R
6. Formenti SC
7. Muggia F
8. Symmans WF
(2001)
Development of tumor-infiltrating lymphocytes in breast cancer after neoadjuvant paclitaxel chemotherapy

Clinical Cancer Research 7:3025–3030.
- PubMed
- Google Scholar
1. Desilva A
2. Wuest M
3. Wang M
4. Hummel J
5. Mossman K
6. Wuest F
7. Hitt MM
(2012)
Comparative functional evaluation of immunocompetent mouse breast cancer models established from PyMT-tumors using small animal PET with [(18)F]FDG and [(18)F]FLT

American Journal of Nuclear Medicine and Molecular Imaging 2:88–98.
- PubMed
- Google Scholar
1. Errington TM
2. Iorns E
3. Gunn W
4. Tan FE
5. Lomax J
6. Nosek BA
(2014) An open investigation of the reproducibility of cancer biology research
eLife 3:e04333.
https://doi.org/10.7554/eLife.04333
- Google Scholar
1. Fu L
2. Kettner NM
(2013) The circadian clock in cancer development and therapy
Progress in Molecular Biology and Translational Science 119:221–282.
https://doi.org/10.1016/B978-0-12-396971-2.00009-9
- PubMed
- Google Scholar
1. Grosso JF
2. Jure-Kunkel MN
(2013)
CTLA-4 blockade in tumor models: an overview of preclinical and translational research

Cancer Immunity 13:5.
- PubMed
- Google Scholar
1. Han X
2. Sterling H
3. Chen Y
4. Saginario C
5. Brown EJ
6. Frazier WA
7. Lindberg FP
8. Vignery A
(2000) CD47, a ligand for the macrophage fusion receptor, participates in macrophage multinucleation
Journal of Biological Chemistry 275:37984–37992.
https://doi.org/10.1074/jbc.M002334200
- PubMed
- Google Scholar
(2016) Study 39: Replication of Willingham, et al., 2012 (PNAS)
Open Science Framework.
https://doi.org/10.17605/OSF.IO/9PBOS
- Google Scholar
1. Hughes P
2. Marshall D
3. Reid Y
4. Parkes H
5. Gelber C
(2007) The costs of using unauthenticated, over-passaged cell lines: how much more data do we need?
BioTechniques 43:575–586.
https://doi.org/10.2144/000112598
- PubMed
- Google Scholar
(2008) Tumor volume in subcutaneous mouse xenografts measured by microCT is more accurate and reproducible than determined by 18F-FDG-microPET or external caliper
BMC Medical Imaging 8:16.
https://doi.org/10.1186/1471-2342-8-16
- PubMed
- Google Scholar
1. Jessy T
(2011) Immunity over inability: The spontaneous regression of cancer
Journal of Natural Science, Biology and Medicine 2:43.
https://doi.org/10.4103/0976-9668.82318
- PubMed
- Google Scholar
1. Kleensang A
2. Vantangoli MM
3. Odwin-DaCosta S
4. Andersen ME
5. Boekelheide K
6. Bouhifd M
7. Fornace AJ
8. Li HH
9. Livi CB
10. Madnick S
11. Maertens A
12. Rosenberg M
13. Yager JD
14. Zhaog L
15. Hartung T
(2016) Genetic variability in a frozen batch of MCF-7 cells invisible in routine authentication affecting cell function
Scientific Reports 6:28994.
https://doi.org/10.1038/srep28994
- PubMed
- Google Scholar
1. Klevorn LE
2. Teague RM
(2016) Adapting Cancer Immunotherapy Models for the real World
Trends in Immunology 37:354–363.
https://doi.org/10.1016/j.it.2016.03.010
- PubMed
- Google Scholar
1. Kokolus KM
2. Capitano ML
3. Lee CT
4. Eng JW
5. Waight JD
6. Hylander BL
7. Sexton S
8. Hong CC
9. Gordon CJ
10. Abrams SI
11. Repasky EA
(2013) Baseline tumor growth and immune control in laboratory mice are significantly influenced by subthermoneutral housing temperature
PNAS 110:20176–20181.
https://doi.org/10.1073/pnas.1304291110
- PubMed
- Google Scholar
1. Krampitz GW
2. George BM
3. Willingham SB
4. Volkmer JP
5. Weiskopf K
6. Jahchan N
7. Newman AM
8. Sahoo D
9. Zemek AJ
10. Yanovsky RL
11. Nguyen JK
12. Schnorr PJ
13. Mazur PK
14. Sage J
15. Longacre TA
16. Visser BC
17. Poultsides GA
18. Norton JA
19. Weissman IL
(2016) Identification of tumorigenic cells and therapeutic targets in pancreatic neuroendocrine tumors
PNAS 113:4464–4469.
https://doi.org/10.1073/pnas.1600007113
- PubMed
- Google Scholar
(1993) Molecular cloning of integrin-associated protein: an immunoglobulin family member with multiple membrane-spanning domains implicated in alpha v beta 3-dependent ligand binding
The Journal of Cell Biology 123:485–496.
https://doi.org/10.1083/jcb.123.2.485
- PubMed
- Google Scholar
1. Liu J
2. Wang L
3. Zhao F
4. Tseng S
5. Narayanan C
6. Shura L
7. Willingham S
8. Howard M
9. Prohaska S
10. Volkmer J
11. Chao M
12. Weissman IL
13. Majeti R
(2015) Pre-clinical Development of a Humanized Anti-CD47 Antibody with Anti-Cancer therapeutic potential
PLoS One 10:e0137345.
https://doi.org/10.1371/journal.pone.0137345
- Google Scholar
1. Macpherson AJ
2. McCoy KD
(2015) Standardised animal models of host microbial mutualism
Mucosal Immunology 8:476–486.
https://doi.org/10.1038/mi.2014.113
- PubMed
- Google Scholar
1. National Research Council (US) Committee for the Update of the Guide for the Care and Use of Laboratory Animals
(2011)
The National Academies Collection: Reports Funded by National Institutes of Health

Guide for the Care and Use of Laboratory Animals, 8th ed, The National Academies Collection: Reports Funded by National Institutes of Health, Washington, National Academies Press.
- Google Scholar
(2005) Immunocompetent mouse model of breast cancer for preclinical testing of EphA2-targeted therapy
Cancer Gene Therapy 12:46–53.
https://doi.org/10.1038/sj.cgt.7700763
- PubMed
- Google Scholar
1. Oldenborg PA
(2004) Role of CD47 in erythroid cells and in autoimmunity
Leukemia & Lymphoma 45:1319–1327.
https://doi.org/10.1080/1042819042000201989
- PubMed
- Google Scholar
(2001) Cd47-Signal Regulatory Protein α (Sirpα) regulates Fcγ and Complement Receptor–mediated Phagocytosis
The Journal of Experimental Medicine 193:855–862.
https://doi.org/10.1084/jem.193.7.855
- Google Scholar
(2000) Role of CD47 as a marker of self on red blood cells
Science 288:2051–2054.
https://doi.org/10.1126/science.288.5473.2051
- PubMed
- Google Scholar
(2001) A murine B cell lymphoma expressing human HER2/ neu undergoes spontaneous tumor regression and elicits antitumor immunity
Cancer Immunology, Immunotherapy 49:649–662.
https://doi.org/10.1007/s002620000155
- Google Scholar
Software
1. R Core Team
(2016)
R: A language and environment for statistical computing

Vienna, Austria.
1. Robinson M
2. Li B
3. Ge Y
4. Ko D
5. Yendluri S
6. Harding T
7. VanRoey M
8. Spindler KR
9. Jooss K
(2009) Novel immunocompetent murine tumor model for evaluation of conditionally replication-competent (oncolytic) murine adenoviral vectors
Journal of Virology 83:3450–3462.
https://doi.org/10.1128/JVI.02561-08
- PubMed
- Google Scholar
1. Saleh F
2. Renno W
3. Klepacek I
4. Ibrahim G
5. Dashti H
6. Asfar S
7. Behbehani A
8. Al-Sayer H
9. Dashti A
10. Kerry C
11. Dashti A
(2005) Direct evidence on the immune-mediated spontaneous regression of human cancer: an incentive for pharmaceutical companies to develop a novel anti-cancer vaccine
Current Pharmaceutical Design 11:3531–3543.
https://doi.org/10.2174/138161205774414556
- PubMed
- Google Scholar
1. Salman T
(2016) Spontaneous tumor regression
Journal of Oncological Science 2:1–4.
https://doi.org/10.1016/j.jons.2016.04.008
- Google Scholar
1. Stevenson GT
(2014) Three major uncertainties in the antibody therapy of cancer
Haematologica 99:1538–1546.
https://doi.org/10.3324/haematol.2013.084640
- PubMed
- Google Scholar
1. Talkington A
2. Durrett R
(2015) Estimating tumor growth rates in vivo
Bulletin of Mathematical Biology 77:1934–1954.
https://doi.org/10.1007/s11538-015-0110-8
- PubMed
- Google Scholar
1. Viechtbauer W
(2010) Conducting Meta-Analyses in R with the metafor Package
Journal of Statistical Software 36:v036.i03.
https://doi.org/10.18637/jss.v036.i03
- Google Scholar
(2004) Spontaneous regression of experimental gliomas--an immunohistochemical and MRI study of the C6 glioma spheroid implantation model
Experimental Neurology 190:478–485.
https://doi.org/10.1016/j.expneurol.2004.08.015
- PubMed
- Google Scholar
1. Willingham SB
2. Volkmer JP
3. Gentles AJ
4. Sahoo D
5. Dalerba P
6. Mitra SS
7. Wang J
8. Contreras-Trujillo H
9. Martin R
10. Cohen JD
11. Lovelace P
12. Scheeren FA
13. Chao MP
14. Weiskopf K
15. Tang C
16. Volkmer AK
17. Naik TJ
18. Storm TA
19. Mosley AR
20. Edris B
21. Schmid SM
22. Sun CK
23. Chua MS
24. Murillo O
25. Rajendran P
26. Cha AC
27. Chin RK
28. Kim D
29. Adorno M
30. Raveh T
31. Tseng D
32. Jaiswal S
33. Enger PØ
34. Steinberg GK
35. Li G
36. So SK
37. Majeti R
38. Harsh GR
39. van de Rijn M
40. Teng NN
41. Sunwoo JB
42. Alizadeh AA
43. Clarke MF
44. Weissman IL
(2012) The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors
PNAS 109:6662–6667.
https://doi.org/10.1073/pnas.1121623109
- PubMed
- Google Scholar
1. Xu JF
2. Pan XH
3. Zhang SJ
4. Zhao C
5. Qiu BS
6. Gu HF
7. Hong JF
8. Cao L
9. Chen Y
10. Xia B
11. Bi Q
12. Wang YP
(2015) CD47 blockade inhibits tumor progression human osteosarcoma in xenograft models
Oncotarget 6:23662–23670.
https://doi.org/10.18632/oncotarget.4282
- PubMed
- Google Scholar
1. Yoshida K
2. Tsujimoto H
3. Matsumura K
4. Kinoshita M
5. Takahata R
6. Matsumoto Y
7. Hiraki S
8. Ono S
9. Seki S
10. Yamamoto J
11. Hase K
(2015) CD47 is an adverse prognostic factor and a therapeutic target in gastric cancer
Cancer Medicine 4:1322–1333.
https://doi.org/10.1002/cam4.478
- PubMed
- Google Scholar
1. Zhang M
2. Hutter G
3. Kahn SA
4. Azad TD
5. Gholamin S
6. Xu CY
7. Liu J
8. Achrol AS
9. Richard C
10. Sommerkamp P
11. Schoen MK
12. McCracken MN
13. Majeti R
14. Weissman I
15. Mitra SS
16. Cheshier SH
(2016) Anti-CD47 Treatment stimulates Phagocytosis of Glioblastoma by M1 and M2 polarized Macrophages and promotes M1 polarized Macrophages in Vivo
PLoS One 11:e0153550.
https://doi.org/10.1371/journal.pone.0153550
- PubMed
- Google Scholar

Decision letter

Joan Massagué

Reviewing Editor; Memorial Sloan-Kettering Cancer Center, United States

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your article "Replication Study: The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors" for consideration by eLife. Your article has been reviewed by four peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Tadatsugu Taniguchi as the Senior Editor. The reviewers have opted to remain anonymous.

In your response before the decision you offered the view that the authors specifically do not indicate whether the results from this replication attempt should be used as evidence for or against the originally reported effect, but rather leave that to the scientific community to discuss and expound upon. However, we note that the whole purpose of the replication study is to provide conclusive evidence for or against the originally reported claims. With inconclusive evidence, the scientific community would have nothing to expound upon.

For publication in eLife the results must be conclusive. Editorially we are willing to consider one last round of revision of your manuscript. However, we want to be clear that eLife will consider a revised manuscript only if it contains enough independent experiments, replicates and internal controls, to arrive at conclusive results. You would need to test therapeutic effects with control tumors that are allowed to grow much larger (e.g. at least 1cm in diameter) than before. You would also need to take into account that responses to immunotherapies in mouse models (and in the clinic) can be delayed.

In sum, you need to provide sufficient experimental evidence to render the replication study conclusive. eLife would consider a revised manuscript only if and when this is achieved.

The original reviews are included below for your consideration.

Reviewer #1:

Errington et al.'s work represents an attempt to reproduce one experiment (out of a series of experiments) originally conducted by Willingham et al. The results of the reproducibility study presented herein not only differ from those of the original, but could also be interpreted to contradict the key findings of Willingham et al. Whether Errington et al.'s results themselves are reproducible remains to be seen, and I argue here that the findings from this group should be taken with a grain of salt.

Reproducibility aside, one necessary condition for conducting a preclinical study comprises the use of a robust model system. Plainly stated, a robust preclinical study wields an ability to yield a reasonable and interpretable result in the face of experimental variability that may be endogenously or exogenously introduced. The results proffered by Errington et al. lend strong credence to the notion that they are not operating with a robust experimental system, thereby leaving me to doubt their results. Specifically, there is tremendous variability in their tumor engraftment, as evinced by their first (and failed) attempt to engraft, followed by their second endeavor in which out of 20 mice, a mere 14 were usable for the study. Moreover, Errington et al. report inconsistent growth kinetics among the 14 mice that were engrafted with the same amount of cells. The fact that Errington et al. seemed to have so much trouble setting up the experiment leads me to question the results emanating from such a study. Simply put, Errington et al.'s study is neither robust due to the extreme variability in their starting points nor reproducible (as they themselves admit), and should not be taken as hardline contrarian evidence against the study by Willingham et al.

In summary, Errington et al. should repeat the study but should first show that they can achieve consistent engraftment of these tumor cells.

Reviewer #2:

Stephen Horrigan and Timothy Errington report their data in the reproducible project related to the paper by Willingham and others in 2012 entitled "The CD 47-signal regulatory protein alpha interaction is a therapeutic target for human solid tumors". The experiment reproduced was a murine clinical trial of an antibody to CD47 for the treatment of breast cancers in a syngeneic orthotopic model. Three parameters were evaluated: Tumor size, infiltration by immune cells into the tumors, and effects on blood counts.

Effects on blood counts, which are considered a toxicity, were reproduced, but the therapeutic effects were not, when compared to the original report. Indeed, in the current reproduction of the experiments, the control IgG was more therapeutically active than the treatment CD47 IgG, although not statistically significant.

There are some concerns in interpreting the therapeutic data.

1) The on target toxicity profile was similar to the original report, which is consistent with the mechanism of action of the CD47 IgG being present in the current model, yet the tumors did not shrink in these mice relative to control. This is an inconsistency. The tumors evaluated were very small. At the time of evaluation of the tumors, 30 days after the treatments began, approximately half of the tumors had weights between 10 and 60 mg, three of the tumors in the control group appeared to be only 10 mg or less in weight. This would correspond to tumors that were only about 2×2×3 mm. With tumors this small I wonder what accuracy of measurement and precision of surgery can be achieved? The methods state that 50,000 cells were injected orthotopically and that the treatment experiment would begin when the tumors were palpable. According to the protocol, the trial was terminated at 30 days. If the tumors were this small after 30 days, how big were the tumors at day 0, at the time of initial palpation? If they were on the order of the same size at that seen 30 days later, this suggests that the tumors grew little in the model, confounding the results; this would be true even in the case of the largest tumors that were observed at one month. This also seems in contrast to the rapid growth of the 50,000 cells injected to reach a "palpable tumor "over the 7-10 days expected, as stated in the Chroscinski registered report that describes the experiments to be done. (This is about a 100x increase in 7-10 days). In addition, there is considerable variability in tumor growth in both groups ranging from five-fold to 20-fold, respectively, between the smallest and the largest tumors. In order to better evaluate these data it would be important to know caliper sizes of all of the tumors at time 0 and any other tumor size measurements that were conducted in the 30 days before the animals were sacrificed. This would allow a better understanding of the growth kinetics and conclusions. Should these issues be discussed in the paper?

2) A second concern is that the authors report that there was a relatively low take rate for this tumor. In the "deviations" section it reports that an initial attempt to inoculate 14 animals with tumor was terminated because of the reduced establishment of tumors. In the "methods" section it states that an initial attempt with 17 mice inoculated with tumor was terminated because of too few animals with tumors. It is not clear why there is a discrepancy in these 2 sites of the paper; or did 2 attempts fail? It also raises the question that if tumor uptake rates were low, and tumor growth rates were slow (as discussed above,) and rates variable (as discussed above,) could this confound the results? These sorts of issues might be discussed in the paper.

Reviewer #3:

This is a well-written paper to replicate the CD47-SIRPa interaction reported by Willingham et al., 2012. The authors have done a nice job of describing their experiments. I recommend accepting this paper after the authors have addressed the comments given below.

Some of the panels in Figure 1 are difficult to read. As a good data reporting practice, I suggest the authors use the observed range of data for the y axis. For example, panel J of Figure 1 shows y-axis from 0 to 80, while the data seem to be between ~50 and ~60. Please use this range instead of 0 to 80. Similar comment applies to each panel in Figure 1.

The text needs to give more insights into Bonferroni correction i.e., how many tests are done and how the p-values were adjusted to get the Bonferroni-corrected p-values.

Given that this is a reproducibility project, the webpage https://osf.io/9pbos/ can provide a better documentation of the data and R codes used to generate all the figures, to obtain Bonferroni corrected p-values, and all the analyses reported. The current documentation is inadequate. For example, the file "Table S4 2 way ANOVA.R" shows R codes for 13 boxplots, but Figure 1 has 15 boxplots. At the top of this file there are objects called "y", "A" and "B", but there is no annotation explaining what is y and what are A and B. Please provide detailed documentation of how each Figure was obtained and for each analysis reported in the paper by providing well-annotated R functions.

The paper lacks a discussion on some thoughts from the authors about potential reasons why their results reported in Figure 2 show an opposite direction to that reported in Figure 6B of Willingham et al., 2012.

Reviewer #4:

This reproducibility project study sought to replicate experiments from Figure 6 A-C and Table S4 of Willingham et al., 2012, showing that treatment of immune competent mice bearing orthotopic M1A2 breast tumors with an anti-CD47 antibody clone MIAP410 led to tumor regression. While the authors of this replication study found that anti-CD47 antibody treatment induced anemia and moderate infiltration of tumors by neutrophils, it had no effect to inhibit tumor growth. Treated tumors on average were actually larger than IgG-treated controls, but there was no statistically significant difference between the two groups. The findings of this study are convincing and unfortunately do not support the major finding from the previous report that treatment with this anti-CD47 antibody induces tumor regression. The key reagents used in this study were well controlled – the cells, for example, were obtained from the author of the original study. As antibody was administered through injection into the fat pad, and no effect on tumor growth was found, it would be of interest to further demonstrate the presence of the antibody within treated tumors, for example by immunostaining tumor sections.

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Replication Study: The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors" for further consideration at eLife. Your revised article has been evaluated by Tadatsugu Taniguchi (Senior Editor), a Reviewing Editor, and two reviewers.

We appreciate the changes you've made and we are moving ahead with acceptance, but we would recommend some further changes to the text first. In particular, we would like the Abstract and the Discussion to reflect the fact that, due to technical issues, the replication study itself is inconclusive. In the Abstract, we would suggest that the sentence that starts "The weight of tumors after 30 days […] " be revised to read as follows:

"However, our efforts to replicate the experiments in the original study which showed that tumor growth was inhibited by anti-CD47 treatment (Figure 6C; Willingham et al., 2012) were inconclusive.

The Discussion should also be revised to make this clear.

Also, the sentence that starts "Both IgG and anti-CD47 treated tumors resulted in[…]", should be revised to make the difference between the replication results and the original results clearer (e.g., please compare the results for lymphocytic infiltrate first, and then compare the results for neutrophilic infiltration).

Please revise your Abstract and Discussion accordingly. We have included the re-reviews below, but you do not need to respond to these comments when you resubmit.

Reviewer #2:

The authors have thoughtfully addressed my major concerns with changes to the data and interpretation, additional figures, additional statistical description. The discussion is more balanced and includes additional caveats to interpretation. With all of the caveats of conducting such a study, drawing definitive conclusions from one study alone (as noted by the authors) is difficult.

Reviewer #4:

While the discussion of the findings has been improved in the revised manuscript, the apparent issues with the tumor model persist as no new experimentation has been added. The previous concerns about the relative inability of the authors to achieve reproducible tumor growth using this model still casts doubt over whether or not the current studies offer solid evidence to contradict the published effects of antiCD47 antibody administration. It seems that for a reproducibility study to present data that is marginally robust, and then to leave the significance of the findings up to the community, this falls short of what should be required for refuting published work, as indicated in the last round of review.

https://doi.org/10.7554/eLife.18173.009

Author response

Our previous comment was: “Finally, we specifically do not indicate whether the results from this replication attempt should be used as evidence for or against the originally reported effect, but rather leave that to the scientific community to discuss and expound upon.” We made this in reference to the replication outcome based on reviewers comment, but this is true of the original study as well. No single study can provide conclusive evidence for or against an effect, but rather it’s the cumulative evidence of multiple experiments and studies that provide the foundation of scientific claims. We outlined this in our article introducing the project. We have added a final paragraph in the Discussion to help clarify what this replication is capable of understanding as well as possible factors, as described in the reviewer comments below, that could impact the outcomes of this study.

For publication in eLife the results must be conclusive. Editorially we are willing to consider one last round of revision of your manuscript. However, we want to be clear that eLife will consider a revised manuscript only if it contains enough independent experiments, replicates and internal controls, to arrive at conclusive results. You would need to test therapeutic effects with control tumors that are allowed to grow much larger (e.g. at least 1cm in diameter) than before. You would also need to take into account that responses to immunotherapies in mouse models (and in the clinic) can be delayed.

In sum, you need to provide sufficient experimental evidence to render the replication study conclusive. eLife would consider a revised manuscript only if and when this is achieved.

This replication attempt, like all of the replication attempts in this project, are designed to perform independent replications with a calculated sample size to detect the originally reported effect size with at least 80% power. Further, this project will report the cumulative evidence across multiple independent replications among multiple studies. Thus, no single replication from this project, just like no original experiment or study, can provide conclusive evidence for or against an effect; rather, it’s the cumulative evidence that forms the foundation of scientific knowledge. However, we understand the desire to perform the experiment independently again, but with modifications to the design outlined and peer reviewed in the Registered Report – before the results were known. While, it’s not within the scope of this project, or as part of this publishing model, to also conduct these studies, the results of this replication bring variables not previously thought to influence the experiment into question (size of the control tumors at the end of the study, length of treatment, etc). Importantly though, it is only because of the results that these and other aspects now become targets for hypothesizing and investigation.

The original reviews are included below for your consideration.

Reviewer #1:

Errington et al.'s work represents an attempt to reproduce one experiment (out of a series of experiments) originally conducted by Willingham et al. [...] In summary, Errington et al. should repeat the study but should first show that they can achieve consistent engraftment of these tumor cells.

We agree the engraftment rate reported in the manuscript was not 100%, however it is unknown to us what the rate was in the original study. The initial attempt was terminated because the prespecified number of animals with detectable tumors to enroll in the study (7 per group) was not achieved, with only 10 out of 14 mice developing tumors. In the second attempt, which inoculated 20 animals based on the observed rate of engraftment in the first attempt, 17 had detectable tumors at the time of randomization, with only 14 mice (with the largest tumor sizes) being enrolled in the study as prespecified in the Registered Report. The remaining 6 mice were used for the baseline hematological measurement, with another one of these developing a tumor before the blood analysis was performed, which means of 20 mice inoculated 18 developed tumors. Variability of tumor volume measurements at the time of assignment were evenly distributed among the two conditions during the randomization. We have revised the manuscript to include the test of tumor volume distribution among the two cohorts.

We agree there is a difference in the range of tumor weights reported in this study, compared to the original work, but there are many factors that could account for discrepancies between the studies. These include factors such as genetic drift and heterogeneity in the cell line population, microbiome of recipient mice, circadian biological responses to therapy, etc., which we can include in the discussion of a revised manuscript. We also agree performing another attempt of this experiment would begin to explore if these, or other, factors influence the outcome of this study. However, what we reported in this manuscript, following the protocol reviewed before these results were known, would be valuable to that effort, whether we or others conducted another attempt. Finally, we specifically do not indicate whether the results from this replication attempt should be used as evidence for or against the originally reported effect, but rather leave that to the scientific community to discuss and expound upon.

In the revised manuscript we have further clarified the engraftment numbers observed in this replication attempt. We have also added a final paragraph in the Discussion to describe possible factors that could impact the outcomes of this study.

Reviewer #2:

There are some concerns in interpreting the therapeutic data.

We agree that presenting the tumor volumes determined from the caliper measurements would be valuable in the manuscript. We generated an additional figure (Figure 2—figure supplement 1) to present this data. We have revised the manuscript to comment on the spread of the reported values observed in this replication attempt in context to the spread of values reported in Willingham et al., 2012. Further, we have revised the manuscript to expand upon other published reports using this model. While we did not find other studies, besides the original study and this replication, performing the assay the same way (orthotopic injections of 50,000 cells), we commented on other in vivo experiments utilizing MT1A2 cells and the approximate growth kinetics (Ahn and Brown, 2008; Noblitt et al., 2005). Interestingly, one paper (Desilva et al., 2012) briefly stated why they purposefully did not utilize this model in their study:

“Furthermore, in our hands spontaneous regression of MT1A2 tumors occurred at a sufficiently high frequency to confound efficacy studies."

The variability in tumor growth could be due to many factors (such as ones described in response to Reviewer #1). The variability should also be viewed in the context that cell growth is exponential under an ideal scenario. There are factors that influence and alter the growth of the tumor initially compared to the continued growth of the tumor in vivo, such as availability of nutrients, oxygen, and space. These points have been included in the Discussion of the revised manuscript.

Thank you for raising this point. There is an error in the methods section when describing the initial attempt to establish tumors in the mice. There was only one initial attempt with 14 mice (as specified in the Registered Report), not 17. This has been corrected in the revised manuscript.

The initial attempt was terminated because the prespecified number of animals with detectable tumors to enroll in the study (7 per group) was not achieved, with only 10 out of 14 mice having tumors. In the second attempt, which increased the number of mice inoculated to 20 based on the observed rate of engraftment in the first attempt, 17 had detectable tumors at the time of randomization. Only 14 mice, with the largest tumor sizes, were enrolled in the study, with the remaining mice used for the baseline hematological measurement. As described in the response to Reviewer #1, we are unaware of what the tumor take rate was in the original study. In the revised manuscript we have further clarified the engraftment numbers observed in this replication attempt. We have also added a final paragraph in the Discussion to describe possible factors that could impact the outcomes of this study.

Reviewer #3:

We have revised Figure 1 to only show the range the observed data fall within as suggested. Part of the reason a wider range was included was to show how the values relate to the normal range (grey region of graph). To also present this aspect of the data, we have revised this data into a table format showing the normal expected range and made it a figure supplement (Figure 1—figure supplement 1).

The text needs to give more insights into Bonferroni correction i.e., how many tests are done and how the p-values were adjusted to get the Bonferroni-corrected p-values.

We agree and have included this in ‘Statistical analysis’ section in the methods of the revised manuscript.

We have increased the annotation and documentation of the files presented online. In regard to the example file, “Table S4 2-way ANOVA.R”, it was used in the power calculations associated with the published Registered Report and contains the original data, not the replication data. We have edited these older files to clearly distinguish the difference. As such, the difference in boxplots is because the original study had 13 parameters, while this replication had 15 parameters due to differences in instrumentation.

We agree and having included a discussion on some of the known variables that were not specifically addressed or could not be easily controlled in this replication attempt.

Reviewer #4:

We agree it would be of interested to further explore if the antibody is detectable within the tumor sections. While we feel this is beyond the scope of this Replication Study, we think making it possible for others interested in examining this or other possible exploratory avenues is worthwhile. We suggest making available the material that can be shared by the lab, in this case the remaining tumor sections that were formalin-fixed, paraffin blocked, and sectioned. We have revised the manuscript to indicate what material is available.

[Editors' note: further revisions were requested prior to acceptance, as described below.]

"However, our efforts to replicate the experiments in the original study which showed that tumor growth was inhibited by anti-CD47 treatment (Figure 6C; Willingham et al., 2012) were inconclusive.

The Discussion should also be revised to make this clear.

Also, the sentence that starts "Both IgG and anti-CD47 treated tumors resulted in […]", should be revised to make the difference between the replication results and the original results clearer (e.g., please compare the results for lymphocytic infiltrate first, and then compare the results for neutrophilic infiltration).

Please revise your Abstract and Discussion accordingly.

We have revised the Abstract and Discussion sections to explicitly state the replication study is confounded by the spontaneous regression observed. The Abstract has also been revised to clearly distinguish the replication and original results for lymphocytic infiltrate and neutrophilic infiltration.

https://doi.org/10.7554/eLife.18173.010

Article and author information

Author details

Stephen K Horrigan

Noble Life Sciences, Gaithersburg, United States

Contribution
SKH, Acquisition of data, Drafting or revising the article

Competing interests
SKH: Noble Life Sciences Inc. is a Science Exchange associated lab.
Reproducibility Project: Cancer Biology

Contribution
RP:CB, Analysis and interpretation of data, Drafting or revising the article
For correspondence
1. tim@cos.io
2. nicole@scienceexchange.com
Competing interests
RP:CB: EI, NP: Employed by and hold shares in Science Exchange Inc.The other authors declare that no competing interests exist.
1. Elizabeth Iorns, Science Exchange, Palo Alto, United States
2. Stephen R Williams, Center for Open Science, Charlottesville, United States
3. Nicole Perfito, Science Exchange, Palo Alto, United States
4. Timothy M Errington, Center for Open Science, Charlottesville, United States

Funding

Laura and John Arnold Foundation

Reproducibility Project: Cancer Biology

The funder had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

The Reproducibility Project: Cancer Biology would like to thank the Weissman lab for sharing critical reagents and data, specifically the MT1A2 cells, the anti-CD47, MIAP410 antibody, and the ELISA data. We thank Frank Graham and McMaster University for providing access to the MT1A2 cells. We would also like to thank the following companies for generously donating reagents to the Reproducibility Project: Cancer Biology; American Type and Tissue Collection (ATCC), Applied Biological Materials, BioLegend, Charles River Laboratories, Corning Incorporated, DDC Medical, EMD Millipore, Harlan Laboratories, LI-COR Biosciences, Mirus Bio, Novus Biologicals, Sigma-Aldrich, and System Biosciences (SBI).

Ethics

Animal experimentation: All animal procedures were approved by the Noble Life Sciences IACUC# 15-05-001SCI and were in accordance with Noble Life Sciences policies on the care, welfare, and treatment of laboratory animals, which adhere to the regulations outlined in the USDA Animal Welfare Act (9 CFR Parts 1,2, and 3) and the conditions specified in the Guide for the Care and Use of Laboratory Animals (National Research Council (US) Committee for the Update of the Guide for the Care and Use of Laboratory Animals, 2011).

Reviewing Editor

Joan Massagué, Memorial Sloan-Kettering Cancer Center, United States

Publication history

Received: May 25, 2016
Accepted: November 21, 2016
Version of Record published: January 19, 2017 (version 1)

Copyright

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.