1. Human Biology and Medicine
Download icon

Registered report: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET

  1. Jake Lesnik
  2. Travis Antes
  3. Jeewon Kim
  4. Erin Griner
  5. Luisa Pedro
  6. Reproducibility Project: Cancer Biology Is a corresponding author
  1. System Biosciences, LLC, United States
  2. Stanford University School of Medicine, United States
  3. University of Virginia, United States
  4. University of Cambridge, United Kingdom
Registered Report
Cited
2
Views
1,464
Comments
0
Cite as: eLife 2016;5:e07383 doi: 10.7554/eLife.07383

Abstract

The Reproducibility Project: Cancer Biology seeks to address growing concerns about reproducibility in scientific research by conducting replications of selected experiments from a number of high-profile papers in the field of cancer biology. The papers, which were published between 2010 and 2012, were selected on the basis of citations and Altmetric scores (Errington et al., 2014). This Registered Report describes the proposed replication plan of key experiments from “Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET” by Peinado and colleagues, published in Nature Medicine in 2012 (Peinado et al., 2012). The key experiments being replicated are from Figures 4E, as well as Supplementary Figures 1C and 5A. In these experiments, Peinado and colleagues show tumor exosomes enhance metastasis to bones and lungs, which is diminished by reducing Met expression in exosomes (Peinado et al., 2012). The Reproducibility Project: Cancer Biology is a collaboration between the Center for Open Science and Science Exchange and the results of the replications will be published in eLife.

https://doi.org/10.7554/eLife.07383.001

Introduction

Exosomes are nanovesicles up to 100 nm in size that are derived from endosomal membranes and secreted by cells as a means of intercellular communication (Mathivanan et al., 2010). They contain a wide array of cargo including proteins, cytokines, and nucleic acids (Kharaziha et al., 2012). Recently, exosomes have been shown to play multiple roles in promoting carcinogenesis, including the regulation of metastatic niche formation, regulation of tumor immune response, and chemotherapeutic resistance (Tickner et al., 2014). Peinado and colleagues reported that exosomes derived from melanoma cells promoted metastasis through education of bone marrow-derived cells in order to prime the pre-metastatic niche and increase vascularization. They further showed that exosome-mediated metastasis was dependent on expression of MET in exosomes, and that MET protein was increased in exosomes found in patients with advanced melanoma (Peinado et al., 2012). MET is an oncogenic receptor tyrosine kinase that promotes proliferation, motility, and migration, and is often aberrantly activated in tumors (Gherardi et al., 2012; Trusolino et al., 2010). These findings indicate that exosomal MET may be a potential therapeutic target or biomarker for metastatic disease.

Supplementary Figure 1C characterizes exosomes isolated from B16-F10 melanoma cells using electron microscopy imaging and Western blotting for exosome protein markers. Supplementary Figure 5A further characterizes these exosomes by assessing the levels of MET and pMET after shRNA-mediated depletion of MET in B16-F10 cells. These figures are essential to reproduce as they validate the expression of key proteins in exosomes that will subsequently be used to replicate Figure 4E. These experiments will be replicated in Protocols 1 and 2.

In Figure 4E, Peinado et al. (2012) reported that reduction of MET protein in melanoma exosomes reduced metastasis of B16-F10 melanoma cells to lung and bone (Peinado et al., 2012). Mice were pre-treated with exosomes isolated from B16-F10 melanoma cells expressing shRNAs directed against Met or control shRNAs, and then B16-F10 cells were injected subcutaneously. Primary tumor growth and metastasis to lungs and bone were assessed by luciferase imaging. This key experiment tests one of the central findings of the paper, namely that MET is necessary for melanoma exosome-mediated promotion of metastasis. This experiment will be replicated in Protocol 3.

Materials and methods

Protocol 1: Lentiviral knockdown of Met or non-silencing control in B16-F10 cells

This protocol utilizes shRNA to knock down Met in B16-F10 cells. This experiment generates key reagents (B16-F10 shMet and B16-F10 shScramble) that will subsequently be used in Protocols 2 and 3.

Sampling

  • Experiment will be conducted once.

    • This experiment will generate B16-F10 shMet and B16-F10 shScramble stable cells.

Note: Information that these are stable transfectants was communicated by authors.

Materials and reagents

ReagentTypeManufacturerCatalog #Comments
B16-F10 cellsCell lineOriginal labn/aFrom original lab
Dulbecco’s modified
Eagle’s medium (DMEM)
Cell cultureVWR45000-30Communicated by authors
Fetal bovine serum (FBS)Cell cultureGE Healthcare (HyClone)SH30088.03Original catalog number
not specified
UltracentrifugeInstrumentSorvalSureSpin 630 rotor
Thick wall ultracentrifuge
tubes
LabwareSpecific brand information will be left up to the discretion of the replicating lab and recorded later
100X Penicillin/streptomycinCell cultureLife Technologies15140-155Communicated by authors
Phosphate buffered
saline (PBS)
Cell cultureFisher ScientificMT-21-040-CMReplaces VWR,
cat# 45000-446;
communicated by authors
TrypsinCell cultureInvitrogen25200-056Original brand not specified
pGIPZ mouse Met shRNA
lentiviral particles
(clone ID: V3LMM_456078)
LentivirusDharmacon/GE LifesciencesVGM5520-200377256Original from Thermo, which
was acquired by Dharmcon/GE
Lifesciences
pGIPZ non-silencing
shRNA lentiviral particles
LentivirusDharmacon/GE LifesciencesRHS4348Original from Thermo,
which was acquired by
Dharmcon/GE Lifesciences
TransDuxCell cultureSystem BiosciencesLV850A-1Replaces polybrene
Fluorescent microscopeInstrumentLeicaDMI300bOriginal instrument not specified
PuromycinCell cultureSigma-AldrichP8833-100MGIncluded during communication
with authors; original brand
not specified
150-mm tissue culture dishLabwareE & K ScientificEK-39160Original brand not specified

Procedure

Note:

  • All cells will be sent for mycoplasma testing and short tandem repeat (STR) profiling.

  • Cells maintained in DMEM supplemented with 10% exosome-depleted FBS, 100 U/ml penicillin and 100 µg/ml streptomycin at 37˚C in a humidified atmosphere at 5% CO2.

  1. Deplete exosomes from FBS by ultracentrifugation at 100,000xg for 70 min at 4˚C.

  2. Transduce B16-F10 cells with pGIPZ mouse Met shRNA (shMet) or pGIPZ non-silencing shRNA (sh Scramble) lentiviral particles following provider standard protocol (see Guide to Lentiviral Packaging and Transduction, System Biosciences) with a multiplicty of infection (MOI) of 10:1 (lentivirus particles:cells) and incubate overnight (16 hr).

    1. Combine culture medium with TransDux to a 1X final concentration.

  3. The next day (16 hr later) replace media.

  4. Determine transduction efficiency by green fluorescent protein (GFP) expression:

    1. Three d after transduction (when efficiency is anticipated to be near 80–90%), use fluorescent microscopy to image cells to determine transduction efficiency based on percent of GFP-positive cells.

      1. Use untransduced B16-F10 cells as GFP-negative cell population.

  5. After determining transduction efficiency, replace media supplemented with 1.5 µg/ml of puromycin to select for transduced cells.

    1. Use untransduced B16-F10 cells as control for puromycin selection treatment.

  6. Continue culture and passage of B16-F10 shMet and B16-F10 shScramble stable cells in puromycin for at least 28 d before further analysis.

    1. After initial selection of 28 d, B16-F10 shMet and B16-F10 shScramble cells should be maintained in puromycin, however when cells are plated for experiments they do not need puromycin added.

    2. Record detailed notes about culturing and passaging of cells, paying particular attention to density.

Deliverables

  • Data to be collected:

    • Detailed notes on cell culturing of both stable cell lines generated.

    • Transduction efficiency as a percentage of GFP-positive cells.

    • Fluorescent microscopy images of GFP-positive cells.

  • Sample delivered for further analysis:

    • B16-F10 shMet and B16-F10 shScramble cells for exosome purification and Western blot analysis of METexpression for Protocol 2.

    • B16-F10 shMet and B16-F10 shScramble cells for exosome purification and mouse injection for Protocol 3.

Confirmatory analysis plan

No analysis performed.

Known differences from the original study

Similar to the original study, the transduction efficiency, determined by GFP expression, and the knockdown efficiency, determined by Western blot, will be measured (Protocol 2). The original protocol for selecting GFP-positive cells included a step to perform fluorescence-activated cell sorting (FACS) of the population to achieve a 95–99% GFP-positive population prior to puromycin selection (information provided by authors). The replication attempt will not include the FACS sorting step and instead will select the stable cells for at least 28 d before further analysis and will maintain the cell lines when not used in experimental procedures under puromycin selection. All known differences are listed in the Materials and reagents section above with the originally used item listed in the Comments section. All differences have the same capabilities as the original and are not expected to alter the experimental design.

Provisions for quality control

The cell lines used in this experiment will undergo STR profiling to confirm their identity and will be sent for mycoplasma testing to ensure there is no contamination. Untransduced B16-F10 cells will be used to confirm the GFP-negative cell population and during puromycin selection to ensure efficient transduction occurs. Detailed cell culture notes will be recorded and made available to monitor growth rates of B16-F10 shScramble and B16-F10 shMet cells. All of the raw data will be uploaded to the project page on the Open Science Framework (OSF) (https://osf.io/ewqzf/) and made publically available.

Protocol 2: Exosome purification and Western blot analysis of MET and phospho-MET expression

This protocol isolates exosomes from B16-F10 shScramble and B16-F10 shMet cells and then utilizes Western blot to characterize protein expression in cells generated in Protocol 1 and exosomes purified from this protocol. MET and pMET protein expression will be determined to verify MET knock down in exosomes, and exosome markers will also be assessed. This experiment will replicate Figures S5A and S1C (right panel).

Sampling

  • Experiment to be repeated a total of three times for a minimum power of 99%.

    • See Power calculations section for details.

  • Each experiment has four cohorts:

    • Cohort 1: B16-F10 shScramble cells

    • Cohort 2: B16-F10 shMet cells

    • Cohort 3: Purified exosomes from B16-F10 shScramble cells

    • Cohort 4: Purified exosomes from B16-F10 shMet cells

  • Each cohort is probed for:

    • HSC70 (exosome marker)

    • TSG101 (exosome marker)

    • CD63 (additional exosome marker)

    • MET

    • pMET Tyr1234/5

    • Glyceraldehyde-3-phosphate dehydrogenase (GAPDH; control)

Materials and reagents

ReagentTypeManufacturerCatalog #Comments
B16-F10 shScramble cellsCell linen/an/aGenerated in Protocol 1
B16-F10 shMet cellsCell linen/an/aGenerated in Protocol 1
Dulbecco’s modified Eagle’s
medium (DMEM)
Cell cultureVWR45000-30Communicated by authors
Fetal bovine serum (FBS)Cell cultureGE Healthcare (HyClone)SH30088.03Original catalog number
not specified
100X Penicillin/streptomycinCell cultureLife Technologies15140-155Communicated by authors
TrypsinCell cultureInvitrogen25200-056Original brand not specified
150 mm tissue culture dishesLabwareE & K ScientificEK-39160Original brand not specified
10 mm tissue culture dishesLabwareFisher Scientific08-772-4FOriginal brand not specified
50 ml centrifuge tubesLabwareFisher Scientific14-959-49AOriginal brand not specified
UltracentrifugeInstrumentSorvalSureSpin 630 rotor
Thick wall ultracentrifuge tubesLabwareSpecific brand information will be left up to the discretion of the replicating lab
and recorded later
Phosphate buffered saline (PBS)Cell cultureFisher ScientificMT-21-040-CMReplaces VWR, cat# 45000-446;
communicated by authors
Radioimmunoprecipitation
assay (RIPA) buffer
 BufferSigma-AldrichR0278Original catalog number
not specified
Complete protease inhibitor
tablets
InhibitorRoche04693116001Original catalog number
not specified
Bicinchoninic acid (BCA) protein
determination kit
Reporter assayPierce23227Original catalog number
not specified
4X Laemmli sample bufferBufferBio-Rad161-0747Original brand not specified
Nanoparticle characterization systemInstrumentNanoSightLM10
Nanoparticle Tracking AnalysisSoftwareNanoSightVersion 2.3Original version not specified
Sodium dodecyl sulfate polyacrylamide
gel electrophoresis (SDS-PAGE)
gradient gels
Western materialsBioRad456-1094Original catalog number
not specified
Pre-stained protein molecular
weight marker
Western materialsBioRad161-0377Original brand not specified
Electrophoresis buffer with SDSBufferBioRad161-0772Original brand not specified
10X Electrophoresis buffer
Tris/glycine/SDS
InstrumentBioRad165-8004Original brand not specified
MethanolChemicalFisher ScientificA412-4Included during communication
with authors; original brand
not specified
Wet transfer systemInstrumentBioRad170-3930Original was a semi-dry system
Immobilon-P polyvinylidene
difluoride (PVDF) membrane
Western materialsMilliporeIPVH10100Communicated by authors
SuperBlockChemicalThermo Scientific37516Original was 2.5% non-fat milk
Tris-buffered saline (TBS)BufferBioRad170-6435Included during communication
with authors; original brand
not specified
Tween-20ChemicalFisher ScientificBP337-500Included during communication
with authors; original brand
not specified
Mouse-anti-HSC70AntibodiesStressgenALX-804-067Dilute 1:500; 70 kDa
Mouse anti-TSG101AntibodiesSanta Cruzsc-7964Dilute 1:500; 45 kDa
Rabbit anti-CD63AntibodiesSystem BiosciencesEXOAB-CD63A-1Dilute 1:1,000; 53 kDa
Mouse anti-METAntibodiesCell Signaling Technology3127Dilute 1:1,000; 145 kDa
Rabbit anti-pMET (Tyr1234/5)AntibodiesCell Signaling
Technology
3077Dilute 1:1,000; 145 kDa
Rabbit anti-GAPDHAntibodiesSanta Cruz Biotechnologysc-25778Dilute 1:1,000; 37 kDa
Sheep anti-mouse-HRPAntibodiesGE HealthcareNA931VDilute 1:4,000–1:20,000
Donkey anti-rabbit-HRPAntibodiesGE HealthcareNA934VDilute 1:4,000–1:20,000
Enhanced chemiluminescent
(ECL) reagent
Western materialsThermo Scientific34095Included during communication
with authors; original brand
not specified
ImageJSoftwareNIHVersion 1.48

Procedure

Note:

  • All cells will be sent for mycoplasma testing and STR profiling.

  • B16-F10 shMet and B16-F10 shScramble stable cells were generated in Protocol 1.

  • All cells maintained in DMEM supplemented with 10% exosome-depleted FBS, 100 U/ml penicillin and 100 µg/ml streptomycin at 37˚C in a humidified atmosphere at 5% CO2.

  • B16-F10 shMet and B16-F10 shScramble stable cells should have 1.5 µg/ml puromycin added to medium while maintaining the cell lines that are not used in the experimental procedure.

  1. Deplete exosomes from FBS by ultracentrifugation at 100,000xg for 70 min at 4˚C.

  2. Grow B16-F10 shScramble and B16-F10 shMet cells for exosome purification and direct lysis.

    1. For exosome purification plate ~5 x 106 cells per 150 mm dish with 25 ml of media. Use two dishes per cell line.

    2. For direct lysis plate ~8–10 x 106 cells per 100 mm dish with 10 ml of media. Use one dish per cell line.

    3. B16-F10 shMet and B16-F10 shScramble cells do not need puromycin added to plates used in the experimental procedure.

  3. Grow exponentially until cells reach 80–90% confluence.

    1. Culture 100 mm dishes overnight.

    2. Culture 150 mm dishes for 48–72 hr.

  4. Directly lyse in 100 mm dishes:

    1. Wash cells in PBS.

    2. Prepare RIPA buffer and harvest cells directly in lysis buffer.

      1. Add a complete protease inhibitor to RIPA buffer.

    3. Clear lysates by benchtop centrifugation at 14,000xg for 20 min at 4˚C.

    4. Measure protein concentration of supernatant with a BCA kit following manufacturer’s instructions.

    5. Prepare 30 µg of total protein per sample with 4X Laemmli buffer.

      1. Store at -20˚C until analysis.

  5. Purify exosomes from 150 mm dishes:

    1. Collect supernatant from cell cultures in 50 ml centrifuge tubes.

    2. Pellet cells from supernatant in a benchtop centrifuge at 500xg for 10 min at 4˚C.

    3. Transfer supernatant to thick wall ultracentrifuge tubes.

    4. Remove cell debris by ultracentrifugation at 20,000xg for 20 min at 4˚C.

    5. Collect supernatant.

    6. Harvest exosomes by ultracentrifugation at 100,000xg for 70 min at 4˚C.

    7. Resuspend the exosome pellet in 20 ml of PBS.

    8. Pellet exosomes by ultracentrifugation at 100,000xg for 70 min at 4˚C.

    9. Resuspend exosome pellet in 100 µl PBS.

      1. Exosomes can be stored at -20˚C for 2–3 weeks.

    10. Measure protein concentration with a BCA kit following manufacturer’s instructions.

      1. Estimated yield of exosomes should be around 1–2 µg/1 x 106 cells.

    11. Characterize exosome pellet using standard Nanosight NTA analysis.

      1. 10–20 µg of exosome protein is needed for analysis.

      2. Report on size distribution and concentration of exosomes.

    12. Prepare 30 µg of total protein per sample with 4X Laemmli buffer.

      1. Store at -20˚C until analysis.

  6. Load 30 µg of total protein with sample buffer in each lane on an SDS-PAGE gel with a protein molecular weight marker.

  7. Run electrophoresis at constant voltage (100 V) for ~1–2 hr in 1X electrophoresis buffer following manufacturer instructions.

  8. Transfer gel to membrane using a wet transfer system at constant amperage (350 mA) for 45 min in 1X Tris-glycine buffer supplemented with 20% methanol.

    1. Pre-soak membrane with methanol and then 1X transfer buffer before assembly.

  9. Block membranes in SuperBlock following manufacturer’s instructions.

  10. Probe membranes with the following primary antibodies overnight at 4˚C diluted in SuperBlock.

    1. mouse anti-HSC70; use at 1:500; 70 kDa

    2. mouse anti-TSG101; use at 1:500; 45 kDa

    3. anti-CD63; use at 1:1000; 53 kDa

    4. mouse anti-MET; use at 1:1000; 145 kDa

    5. rabbit anti-pMET Tyr1234/5; use at 1:1000; 145 kDa

    6. rabbit anti-GAPDH; use at 1:1000; 37 kDa

  11. Wash membranes three times 10 min in Ticket Tax Box Service (TTBS).

    1. TTBS = 1X TBS supplemented with 0.1% tween.

  12. Detect primary antibodies with the following secondary antibodies diluted in SuperBlock for 1 hr.

    1. sheep anti-mouse-HRP; use at 1:4000–1:20,000

    2. donkey anti-rabbit-HRP; use at 1:4000–1:20,000

  13. Wash membranes three times for 10 min each in TTBS.

  14. Detect using ECL reagent following manufacturer’s instructions.

  15. Quantify intensities of the immunoreactive bands by densitometry.

    1. Normalize total MET to GAPDH.

    2. Normalize pMET (Tyr1234/1235) to GAPDH.

  16. Repeat independently two additional times.

Deliverables

  • Data to be collected:

    • Exosome characterization data (including protein concentration using a BCA kit).

    • Images of probed membranes (full images with ladder) (compare to Figure S1C [right panel]).

    • Quantified levels of total MET and Phospho-MET normalized to GAPDH for all conditions (compare to Figure S5A).

Confirmatory analysis plan

This experiment assesses if knockdown of Met alters total MET and phospho-MET expression in cells and exosomes. This replication attempt will perform the following statistical analysis:

  • Statistical analysis:

    • One-way multivariate analysis of variance (MANOVA) comparing the relative mean of photon signal for total MET and phospho-MET normalized to GAPDH from B16-F10 shScramble and B16-F10 shMet cells.

      • Planned comparisons with the Bonferroni correction:

        1. Total MET in B16-F10 shScramble cells compared with B16-F10 shMet cells

        2. Phospho-MET in B16-F10 shScramble cells compared with B16-F10 shMet cells

    • One-way MANOVA comparing the relative mean of photon signal for total MET and phospho-MET normalized to GAPDH from B16-F10 shScramble and B16-F10 shMet exosomes.

      • Planned comparisons with the Bonferroni correction:

        1. Total MET in B16-F10 shScramble exosomes compared to B16-F10 shMet exosomes

        2. Phospho-MET in B16-F10 shScramble exosomes compared to B16-F10 shMet exosomes

  • Meta-analysis of effect sizes:

    • Compute the effect sizes of each comparison, compare them against the reported effect size in the original paper, and use a meta-analytic approach to combine the original and replication effects, which will be presented as a forest plot.

Known differences from the original study

The cell lines when not used in experimental procedures will be maintained under puromycin selection. The replication attempt will include an additional exosome marker, CD63, not included in the original study. All known differences are listed in the Materials and reagents section above with the originally used item listed in the Comments section. All differences have the same capabilities as the original and are not expected to alter the experimental design.

Provisions for quality control

The cell lines used in this experiment will undergo STR profiling to confirm their identity and will be sent for mycoplasma testing to ensure there is no contamination. This protocol analyzes the knockdown efficiency of c-Met in B16-F10 cells. Isolated exosomes are characterized by NanoSight and are analyzed for typical exosome markers by Western blot, including CD63, an additional marker not included in the original study. All of the raw data, including the image files and quantified bands from the Western blot, will be uploaded to the project page on the OSF (https://osf.io/ewqzf/) and made publically available.

Protocol 3: Exosome-dependent MET signaling on primary tumor growth and metastasis

This experiment tests the effect of exosome-derived MET on primary growth and metastasis of melanoma cells. This is a replication of Figure 4E, which assesses metastasis in lungs and bone using bioluminescent imaging.

Sampling

  • Experiment will use seven mice per cohort for a minimum power of 82%.

    • See Appendix for detailed power calculations

  • Each experiment has three cohorts:

    • Cohort 1: Synthetic unilamellar liposomes injected into C57BL/6 mice

    • Cohort 2: B16-F10 shScramble exosomes injected into C57BL/6 mice

    • Cohort 3: B16-F10 shMet exosomes injected into C57BL/6 mice

Materials and reagents

ReagentTypeManufacturerCatalog #Comments
B16-F10 shScramble cellsCell linen/an/aGenerated in Protocol 1
B16-F10 shMet cellsCell linen/an/aGenerated in Protocol 1
Dulbecco’s modified Eagle’s
medium (DMEM)
Cell cultureVWR45000-30Communicated by authors;
for B16-F10 shScramble
and shMet cells
DMEMCell cultureLife Technologies11995-040Communicated by authors;
for B16-F10-luciferase cells
Fetal bovine serum (FBS)Cell cultureGE Healthcare (HyClone)SH30088.03Original catalog number
not specified
100X Penicillin/streptomycinCell cultureLife Technologies15240-062Communicated by authors
150 mm tissue culture dishLabwareE & K ScientificEK-39160Original brand not specified
10 mm tissue culture dishLabwareFisher Scientific08-772-4FOriginal brand not specified
T75 tissue culture flasksLabwareSigma-AldrichCLS430641Originally not specified
T25 tissue culture flasksLabwareSigma-AldrichC6356Originally not specified
50 ml centrifuge tubesLabwareFisher Scientific14-959-49AOriginal brand not specified
UltracentrifugeInstrumentSorvalSureSpin 630 rotor
Thick wall ultracentrifuge tubesLabwareSpecific brand information will be left up to the discretion of the replicating lab
and recorded later
Phosphate buffered saline (PBS)BufferFisher Scientific or
Life Technologies
MT-21-040-CM or 70011-044Replaces VWR, cat# 45000-446
Communicated by authors
TrypsinCell cultureInvitrogen or
Life Technologies
25200-056 or 12604-021Original brand not specified
Bicinchoninic acid (BCA)
protein determination kit
Reporter assayPierce23227Original catalog number
not specified
Nanoparticle characterization
system
InstrumentNanoSightLM10
Nanoparticle Tracking AnalysisSoftwareNanoSightVersion 2.3Original version not specified
Synthetic unilamellar
100 nm liposomes
ChemicalEncapsula
NanoSciences
n/aComposition: 13 mg/ml
L-α-phosphatidylcholine,
2.78 mg/ml cholesterol
(7:3 molar ratio P:C);
communicated by authors
6-week-old C57BL/6 female miceAnimal modelCharles RiverStrain code: 027Replaces Jackson Laboratories used
in the original study; age
communicated by authors
B16-F10-luciferase cellsCell lineOriginal labn/aFrom original lab
1 ml syringe with Luer-Lok tipLabwareFisher Scientific14-823-30Original brand not specified
NeedleLabwareSpecific brand information will be left up to the discretion of the replicating lab
and recorded later
3/10 cc syringe with 29G1/2
attached needle
LabwareTerumo Medical
Products
SS30M2913Original brand not specified
AErrane (Isoflurane)ChemicalBaxtern/aReplaces original from Baxter,
catalog # 400-326-09;
communicated by authors
OxygenChemicalPraxairTC 3AAM 154Replaces original from Airgas;
communicated by authors
D-luciferin, potassium saltReporter assayBiosynthL-8220Replaces original brand from
Life Technologies;
communicated by authors
IVIS Spectrum systemInstrumentXenogen (Caliper)
Living Image softwareSoftwareXenogen (Caliper)Version 4.2

Note:

  • All cells will be sent for mycoplasma testing and STR profiling.

  • B16-F10 shMet and B16-F10 shScramble stable cells were generated in Protocol 1.

  • All cells maintained in DMEM supplemented with 10% exosome-depleted FBS, 100 U/ml penicillin and 100 µg/ml streptomycin at 37˚C in a humidified atmosphere at 5% CO2.

  • B16-F10 shMet and B16-F10 shScramble stable cells should have 1.5 µg/ml puromycin added to medium while maintaining the cell lines that are not used in the experimental procedure.

  • The original study indicated mice were 8–10 weeks old, however the authors clarified that the mice were 6 weeks old.

  1. Deplete exosomes from FBS by ultracentrifugation at 100,000xg for 70 min at 4˚C.

  2. Two–three days before needing exosomes, plate ~5 x 106 B16-F10 shScramble and B16-F10 shMet cells per 150 mm dish with 25 ml of media for exosome purification.

    1. Use three to six dishes per cell line to obtain needed amount of exosomes.

    2. B16-F10 shMet and B16-F10 shScramble cells do not need puromycin added to plates used in the experimental procedure.

  3. Culture cells exponentially for 48–72 hr until cells reach 80–90% confluence.

  4. Purify exosomes from 150 mm dishes:

    1. Collect supernatant from cell cultures in 50 ml centrifuge tubes.

    2. Pellet cells from supernatant in a benchtop centrifuge at 500xg for 10 min at 4˚C.

    3. Transfer supernatant to thick wall ultracentrifuge tubes.

    4. Remove cell debris by ultracentrifugation at 20,000xg for 20 min at 4˚C.

    5. Collect supernatant.

    6. Harvest exosomes by ultracentrifugation at 100,000xg for 70 min at 4˚C.

    7. Resuspend the exosome pellet in 20 ml of PBS.

    8. Pellet exosomes by ultracentrifugation at 100,000xg for 70 min at 4˚C.

    9. Resuspend exosome pellet in 100 µl PBS.

      1. An aliquot of the exosome preparation can be stored at -20˚C until Nanosight analysis.

    10. Measure protein concentration with a BCA kit following manufacturer’s instructions.

      1. Estimated yield of exosomes should be around 1–2 µg/1x106 cells.

    11. Characterize exosome pellets using standard Nanosight NTA analysis.

      1. An aliquot of each exosome preparation will be stored at -20˚C and analyzed all at once following the final preparation.

      2. 10–20 µg of exosome protein is needed for analysis.

      3. Report on size distribution and concentration of exosomes.

      4. Report the number of exosomes per µg protein (as measured by BCA) for each sample.

    12. Prepare samples at a concentration of 50 ng/µl to achieve 5 µg of total protein diluted in 100 µl of PBS.

  5. Following protein quantification of each preparation, inject intravenously, via retro-orbital injection, freshly isolated B16-F10 shScramble or B16-F10 shMet exosomes, or synthetic unilamellar 100 nm liposomes into 6-week-old C57BL/6 female mice three times a week for a total of 28 d.

    1. Sample injection schedule:

      1. Each cohort will be injected on Monday, Wednesday, and Friday with a fresh exosome preparation (>35 µg total) each week for a total of 4 weeks.

      2. Step 4 will be performed for each injection day.

    2. It is crucial to inject fresh exosomes every time, do not freeze down, and inject right after purification and quantification following steps 2–4 above.

    3. Volume of injection is 100 µl.

    4. Amount of synthetic liposomes injected, 1.25 µg of L-α-phosphatidylcholine (PC) will mimic 5 µg of exosome protein, which is based on a theoretical 4:1 protein:PC ratio (communicated by authors).

      1. Dilute 1.92 µl of a 1:20 dilution of the stock concentration of synthetic liposomes into 100 µl PBS for each injection.

  6. After 28 d, inject 1 x 106 B16-F10-luciferase cells diluted in 100 µl of PBS subcutaneously in the flank of mice.

  7. Measure primary tumor volume three times a week for a total of 21 d.

    1. Use calipers to measure width and height with volume determined as (length x width2)/2. (additional recorded information)

      1. Note: tumor volume detection will be limited to <1000 mm3.

    2. Record latency. (additional recorded information)

      1. Note: Perform Steps 8 through 10 (luciferin injection, euthanasia, dissection, and imaging) from mice from different cohorts in parallel (i.e., one from each of the three cohorts) so variation during the procedure is equal across cohorts. (additional detail)

  8. After 21 d anesthetize mice and inject 50 mg/kg of D-luciferin via retro-orbital injection in 100–200 µl PBS (volume depending on the body weight).

    1. Weigh mice.

    2. Use isoflurane and O2 to anesthetize mice.

  9. Five min later euthanize mice by cervical dislocation under anesthesia and dissect tissues (lungs and femurs).

    1. Dissect out primary tumors and record weight (additional parameter).

  10. Image dissected primary lungs and bones (femurs) for luciferase expression in IVIS Imaging system

    1. Record the time from euthanasia to imaging for each mouse.

    2. Record photon flux.

      1. Take two–three exposure times for each sample.

        1. Use same exposure time for each tissue from all mice during analysis.

Deliverables

  • Data to be collected:

    • Mouse health records (injection schedule, time from tumor cell injection to detectable tumors [latency], weight of mice at end of experiment, mortality report)

    • Exosome characterization data (including protein concentration using a BCA kit).

    • Raw numbers and calculated tumor volume for all mice, and graph of tumor volume versus time for all conditions during course of treatment.

    • Time of euthanasia to imaging for each mouse.

    • Images of lungs and bones for luciferase expression (compare to Figure 4E).

    • Raw photon flux values of all analyzed images of each tissue for all conditions using the same exposure time (compare to Figure 4E).

    • Final weight of tumors.

Confirmatory analysis plan

This experiment assesses if knockdown of Met alters primary tumor growth and metastasis. This replication attempt will perform the following statistical analysis:

  • Statistical Analysis:

    • Bonferroni corrected one-way ANOVAs comparing lung photon flux of mice treated with synthetic unilamellar liposomes, or exosomes from B16-F10 shScramble or B16-F10 shMet cells and the following planned comparisons with the Fisher’s least significant difference (LSD) test:

      1. Synthetic unilamellar liposomes compared to B16-F10 shScramble exosomes

      2. B16-F10 shMet exosomes compared to B16-F10 shScramble exosomes

    • Bonferroni corrected Kruskal–Wallis test comparing bone photon flux of mice treated with synthetic unilamellar liposomes, or exosomes from B16-F10 shScramble or B16-F10 shMet cells and the following planned comparisons (Wilcoxon–Mann–Whitney test) with the Fisher’s LSD test:

      1. Synthetic unilamellar liposomes compared to B16-F10 shScramble exosomes

      2. B16-F10 shMet exosomes compared to B16-F10 shScramble exosomes

  • Meta-analysis of effect sizes:

    •  Compute the effect sizes of each comparison, compare them against the reported effect size in the original paper and use a meta-analytic approach to combine the original and replication effects, which will be presented as a forest plot.

  • Exploratory analysis:

    •  Comparison of primary tumor growth rates.

      • This is exploratory analysis. We will measure tumor growth rates across all mouse cohorts over the length of the study. These data were collected, but not reported in the original study, and found to not be different. We will plot growth curves and calculate area under the curve for each mouse. We will then perform a one-way analysis of variance (ANOVA) analysis, with the following planned comparisons with the Fisher’s LSD test:

        1. Synthetic unilamellar liposomes compared to B16-F10 shScramble exosomes

        2. B16-F10 shMet exosomes compared to B16-F10 shScramble exosomes

    • Comparison of final primary tumor weights.

      • This is exploratory analysis. We will measure tumor weights across all mouse cohorts at the end of the study. These data were not reported in the original study. We will perform a one-way ANOVA analysis, with the following planned comparisons with the Fisher’s LSD test:

        1. Synthetic unilamellar liposomes compared to B16-F10 shScramble exosomes

        2. B16-F10 shMet exosomes compared to B16-F10 shScramble exosomes

Known differences from the original study

The cell lines when not used in experimental procedures will be maintained under puromycin selection. The number of exosomes injected (based on protein content) will be reported for each preparation from each cohort. The original data on primary tumor growth were not shown and final primary tumor weights were not recorded. The replication attempt will record and present these data as well as tumor latency. All known differences are listed in the Materials and reagents section above with the originally used item listed in the Comments section. All differences have the same capabilities as the original and are not expected to alter the experimental design.

Provisions for quality control

The cell lines used in this experiment will undergo STR profiling to confirm their identity and will be sent for mycoplasma testing to ensure there is no contamination. Isolated exosomes will be injected immediately after protein quantification and will be characterized by NanoSight following the final preparation to ensure the integrity of the samples. Exosomes and luciferin will be injected intravenously, via retro-orbital injection, similar to the original study. While it will be attempted to be the same for all animals, the time from euthanasia to imaging for each mouse will be recorded as an additional quality control parameter. All of the raw data, including the image files and photo flux values, will be uploaded to the project page on the OSF (https://osf.io/ewqzf/) and made publically available.

Power calculations

For additional details on power calculations, please see analysis scripts and associated files on the OSF: 

https://osf.io/nyb8d/

Protocol 1

Not applicable

Protocol 2

Summary of original data reported in Figure S5A (calculated from data shared by original authors):

Dataset being analyzedMeanSDN
Total METB16-F10 shScramble cells1.0000.00852
B16-F10 shMet cells0.6410.02542
Phospho-METB16-F10 shScramble cells1.0000.03182
B16-F10 shMet cells0.2340.01342

The replication will also include analysis of total MET and phospho-MET in exosomes from B16-F10 shScramble and B16-F10 shMet cells. We will use the same calculated sample size as determined for the cells with the assumption that the cells and exosomes have similar values, which Peinado and colleagues (2012) present in Figure 4A.

Test family

  • two-tailed t test, difference between two independent means, Bonferroni’s correction: alpha error = 0.025

Power calculations performed with G*Power software, version 3.1.7 (Faul et al., 2007).

Total Met:

Group 1Group 2Effect size dA priori powerGroup 1
sample size
Group 2
sample size
B16-F10 shScrambleB16-F10 shMet18.9736799.9%2*2*
  1. * A minimum sample size of three per group will be used making the power 99.9%

Phospho-Met:

Group 1Group 2Effect size dA priori powerGroup 1
sample size
Group 2
sample size
B16-F10 shScrambleB16-F10 shMet31.3841199.9%2 2
  1.  A minimum sample size of three per group will be used making the power 99.9%.

Protocol 3

Summary of original data reported in Figure 4E (calculated from data shared by original authors):

Dataset being analyzedMeanSDN
Lung photon fluxSynthetic unilamellar liposomes23,3869,1387
Exosomes from B16-F10 shScramble cells50,77114,9667
Exosomes from B16-F10 shMet cells12,6679,0327
Bones photon fluxSynthetic unilamellar liposomes005
Exosomes from B16-F10 shScramble cells44,66032,5955
Exosomes from B16-F10 shMet cells005

Lung photo flux:

Test family

  • ANOVA: Fixed effects, omnibus, one-way, Bonferroni’s correction: alpha error = 0.025.

Power calculations performed with G*Power software, version 3.1.7 (Faul et al., 2007).

ANOVA F-test statistic performed with R software, version 3.1.2 (Team, 2014).

Partial η2 calculated from (Lakens, 2013).

DatasetGroupsF-test statisticPartial η2Effect
size f
A priori
power
Total sample
size
LungSynthetic unilamellar
liposomes,
B16-F10 shScramble,
B16-F10 shMet
F(2,18) = 20.84170.698411.5217694.8%*12 total mice*
(3 groups)
  1. * 21 total mice will be used based on the bones photon flux planned comparison calculations making the power 99.9%.

Test family

  • 2 tailed t test, difference between two independent means, Fisher’s LSD correction: alpha error = 0.025.

Power calculations performed with G*Power software, version 3.1.7 (Faul et al., 2007).

Group 1Group 2Effect
size d
A priori
power
Group 1
sample size
Group 2
sample size
Synthetic unilamellar
liposomes
B16-F10 shScramble2.2086186.1%6*6*
B16-F10 shScrambleB16-F10 shMet3.0827787.6%44
  1. * 7 mice will be used per group based on the bones photon flux calculations making the power 92.5%.

  2. 7 mice will be used per group based on the bones photon flux calculations making the power 99.8%.

Bones photon flux:

Test family

  • ANOVA: Fixed effects, omnibus, one-way, Bonferroni’s correction: alpha error = 0.025.

Power calculations performed with G*Power software, version 3.1.7 (Faul et al., 2007). ANOVA F-test statistic performed with R software, version 3.1.2 (Team, 2014). Partial η2 calculated from (Lakens, 2013).

DatasetGroupsF-test statisticPartial η2Effect size fA priori
power
Total
sample size
BonesSynthetic unilamellar
liposomes,
B16-F10 shScramble,
B16-F10 shMet
F(2,12) = 9.38660.610051.2507782.2%* 12*
(3 groups)
  1. * Since the nonparametric Kruskal–Wallis test will be performed for the analysis instead of an ANOVA, a 15% adjustment in sample size (15 instead of 12) is taken into account. A total of 21 mice will be used based on the bones photon flux planned comparison calculations making the power 98.1% (using a 15% adjusted sample size of 18 instead of 21).

Test family

  • 2 tailed t test, Wilcoxon–Mann–Whitney test (two groups), Fisher’s LSD correction: alpha error = 0.025.

Power calculations performed with G*Power software, version 3.1.7 (Faul et al., 2007).

Group 1Group 2Effect size dA priori powerGroup 1
sample size
Group 2
sample size
Synthetic unilamellar
liposomes
B16-F10 shScramble1.9376981.6%77
B16-F10 shScrambleB16-F10 shMet1.9376981.6%77

References

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
    A language and environment for statistical computing
    1. R Development Core Team R
    (2014)
    R Foundation for Statistical Computing., http://www. R-project.org/.
  9. 9
  10. 10

Decision letter

  1. Michael R Green
    Reviewing Editor; Howard Hughes Medical Institute, University of Massachusetts Medical School, United States

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your work entitled "Registered report: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET" for peer review at eLife. Your submission has been favorably evaluated by Charles Sawyers (Senior Editor), Michael Green (Reviewing Editor), and four reviewers, one of whom, John Harris, has agreed to reveal his identity.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

Although the reviewers were generally positive there were several substantive concerns that need to be addressed in a revised registered report. Of particular concern is the method used for exosome isolation (point 1) and the apparent conflict of interest (point 2).

Essential revisions:

1) Given the aim of this project, it is of paramount importance that all details of the experimental design be reproduced as described in Peinado et al., Nature Medicine, 2012. In particular, there are some critical differences in the experimental procedures of the proposed study that may lead to big differences in the results compared to the original Peinado et al. study. First and foremost, the method proposed to isolate exosomes for western blot and functional studies, as well as for the depletion of exosomes from FBS is based on ExoQuick extraction rather than sequential ultracentrifugation as in the original publication. This is a major issue that can hinder the reproducibility of the data, as during exosome isolation using ExoQuick, contaminants (e.g. VEGF) are trapped by ExoQuick, therefore education would not be strictly and solely due to exosomes. Clearly the authors are aware of this issue, as they have highlighted this difference. The main problem is that 1 μg of exosomal protein from ExoQuick-isolated exosomes would be the equivalent of 0.1-0.5 μg of exosomes isolated by classical ultracentrifugation (UC), as in the Peinado et al. study. This 10-fold difference in the amount of exosome protein delivered in education studies would lead to "under-education". Since in the Peinado et al. study the exosomal preparations were free of contaminants, as demonstrated by electron microscopy and western blot, and the authors' likely carry significant contaminants, a side by side comparison of ExoQuick and ultracentrifugation would be required to determine the amount of ExoQuick exosomal preparation equivalent to 10 μg of exosomes. A reference is not sufficient, actual data needs to be presented for this comparison, including quantitation and electron microscopy studies. The authors would have to measure the yield of exosomes, proteins and contaminants from both methods in parallel.

There are numerous publications presenting data comparing exosomal isolation methods that support the use of ultracentrifugation versus polymer-based precipitating methods such as ExoQuick. In particular, three papers demonstrate that up to 23 times more material is precipitated with ExoQuick compared to standard serial UC (Van Deun J et al., Journal of Extracellular Vesicles, 2014, 18(3), PMID: 25317274; Zlotogorski-Hurvitz A et al., J Histochem Cytochem. 2015, 63(3): 181-9, PMID: 25473095; Yamada T et al., J Vet Med Sci. 2012, 74(11):1523-5, PMID: 22785357). A recent study provides electron microscopy pictures that demonstrate that the higher protein yield obtained with the ExoQuick precipitation method compared to the UC gold standard is due to contaminants such as large protein aggregates (Sáenz-Cuesta M et al., Front Immunol, 2015, 6:50, PMID: 25762995).

2) Importantly, the fact that the first three authors on the proposed Reproducibility project study are employees of System Biosciences, the company that produces ExoQuick as well as ExoFBS raises a red flag and creates a huge conflict on interest in terms of the choice of methodology. The choice of experimental approaches and reagents should be driven by the desire to perform the tests in conditions as close as possible to the original study, but the authors are clearly choosing a critical reagent that introduces a large variation from the original study based on financial interests.

3) It is not clear whether the authors plan to use fresh or frozen (-20°C) exosomal preparations for education. There have been studies that have shown that freezing exosomes results in lysis of more than 50% of the preparation (Oosthuyzen et al., J Physiol. 2013, 591(Pt 23): 5833-5842) which will affect the amount delivered in a single dose (if the exosomes were quantified prior to freezing and that amount is used for calculation of material to inject for education). Freezing could also affect the functionality of exosomes. The original Peinado et al. study had performed the education studies with fresh exosomes.

4) Another major problem is that the authors are using non-transduced B16-F10 cells as control for the B16-F10 shMET. It is highly recommended that a scrambled short hairpin control (shScramble) be used, as it is widely known that infection and puromycin treatment of cells may alter the cell population and the appropriate scrambled control in the same vector is available from the same company that produces the shMET. Importantly, the authors plan to rely solely on puromycin resistance to select the knockdown clones, but this is not sufficient to isolate a pure population of transduced cells that maintains high levels of vector integration and expression. In addition to verifying knockdown of MET, the authors also need to show that Met knockdown did not affect the growth of the cells (every integration event is unique and you never know where the lentivirus integrated).

5) In protocol 1: The authors propose to use puromycin selection to generate stable B16-F10 shMet transfectants instead of FACS sorting GFP-positive cells, which should be sufficient. Upon reviewing the original paper, we could not find any details how the authors generated B16-F10 shMet cells, and whether these were stable transfectants. Was this communicated directly by the authors of the study?

6) It looks like they never tested whether MET was downregulated in exosomes themselves in supplementary Figure 5a, just in the parent cells from which they purified the exosomes. This is planned for the B16-F10 control and shMET exosomes, which we think is good. We would also suggest including B16-F1 exosomes, as there was a comparison by Western blot for the B16-F10 to B16-F1 exosomes in Figure 4a, at least that would be something to compare across studies.

7) In the original paper there is a microarray analysis where genes with a fold change greater than 2 are reported. Apparently p-values from a t-test were computed with a permutation approach but not reported. It would be interesting to repeat also this experiment reporting the statistical significant genes controlling the false discovery rate.

8) In the statistical analyses plan of the report it is claimed that Fisher's LSD correction has been used. However the alpha error 0.025 reported is due to a Bonferroni's correction; indeed the Fisher's LSD correction used by the authors is not taking into account that multiple comparisons will be performed and it's good only for the calculation of the effect size (see Anthony J. Hayter. The maximum familywise error rate of fisher's least significant difference test. Journal of the American Statistical Association, 1986, 81(396):

1000-1004, doi: 10.1080/01621459.1986.10478364).

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Registered report: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET" for further consideration at eLife. Your revised article has been evaluated by Charles Sawyers (Senior Editor), Michael Green (Reviewing Editor), and four reviewers, one of whom, John Harris, has agreed to reveal his identity.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

The reviewers felt that the revised manuscript had been considerably improved. However, two major problems remain. First, given the aim of this project, it is of paramount importance that all details of the experimental design be reproduced as described in Peinado et al., Nature Medicine, 2012. Therefore, all of the reviewers feel strongly that exosome purification needs to be done by sequential ultracentrifugation as in Peinado et al. (see point 1). Second, the conflict of issue has not been adequately resolved (see point 2).

Essential revisions:

1) We appreciate the authors' efforts to address my concerns. However, the new experiment (protocol 3) added to the manuscript proposing to compare ExoQuick extraction with the sequential ultracentrifugation presented in the original publication, is not appropriate for the "Reproducibility project". This is a new experiment that can be the focus of a new manuscript. Even for such a method comparison manuscript, to indeed validate their side-by-side comparison of ExoQuick and ultracentrifugation, the authors should perform unbiased analyses, specifically proteomic analysis of exosomes by mass spectrometry, not just western blot analyses. For the purpose of reproducing the data in the original Peinado et al. publication, the original exosome purification method should be used. Despite the fact that the differences from the original study are clearly stated for each protocol, for Protocols 2 and 4, the authors are still relying ONLY on ExoQuick for exosome purification. Given that during exosome isolation using ExoQuick, contaminants (e.g. VEGF) are trapped by ExoQuick, and the fact that education would not be strictly and solely due to exosomes, we believe that all the experiments proposed in this paper should be performed using the sequential ultracentrifugation, not ExoQuick. Moreover, the authors are using System Biosciences ExoFBS for all the experiments, rather than depleting exosomes from the FBS using sequential ultracentrifugation.

2) The authors declared their conflict of interest as employees of System Biosciences, the company that produces ExoQuick as well as ExoFBS. However, the determination of the authors to use System Biosciences reagents to conduct this study (even in the context of comparing methods) is impacted by the conflict of interest and driven by their desire to promote System Biosciences products. Therefore, because of their vested interest, employees of SBI should be excluded from the author list of this reproducibility project or the manuscript should not use any System Biosciences products.

3) The authors included the scrambled short hairpin control (shScramble) B16-F10 cells as s control for the B16-F10 shMET experiments. However, the authors plan to rely solely on puromycin resistance to select the knockdown clones, but this is not sufficient to isolate a pure population of transduced cells that maintains high levels of vector integration and expression. The authors should evaluate the knockdown after 28 days of puromycin selection. The authors should either use fluorescence-activated cell sorting to select a pure population or they should maintain the cells under puromycin selection. Just as another control, it would be desirable to compare both B16-F10 wild-type and Scramble in order to verify that the scramble does not have any undesirable effects, since infection can cause genome instability that way would be a perfect control.

4) The synthetic liposome control was a bit confusing. The authors say they will use 5 µg to match the protein on exosomes, but there should not be any protein on synthetic liposomes. We think they should figure out a vesicle concentration based on NanoSight and then use the same concentration of synthetic liposomes.

https://doi.org/10.7554/eLife.07383.002

Author response

1) Given the aim of this project, it is of paramount importance that all details of the experimental design be reproduced as described in Peinado et al., Nature Medicine, 2012. In particular, there are some critical differences in the experimental procedures of the proposed study that may lead to big differences in the results compared to the original Peinado et al. study. First and foremost, the method proposed to isolate exosomes for western blot and functional studies, as well as for the depletion of exosomes from FBS is based on ExoQuick extraction rather than sequential ultracentrifugation as in the original publication. This is a major issue that can hinder the reproducibility of the data, as during exosome isolation using ExoQuick, contaminants (e.g. VEGF) are trapped by ExoQuick, therefore education would not be strictly and solely due to exosomes. Clearly the authors are aware of this issue, as they have highlighted this difference. The main problem is that 1 μg of exosomal protein from ExoQuick-isolated exosomes would be the equivalent of 0.1-0.5 μg of exosomes isolated by classical ultracentrifugation (UC), as in the Peinado et al. study. This 10-fold difference in the amount of exosome protein delivered in education studies would lead to "under-education". Since in the Peinado et al. study the exosomal preparations were free of contaminants, as demonstrated by electron microscopy and western blot, and the authors' likely carry significant contaminants, a side by side comparison of ExoQuick and ultracentrifugation would be required to determine the amount of ExoQuick exosomal preparation equivalent to 10 μg of exosomes. A reference is not sufficient, actual data needs to be presented for this comparison, including quantitation and electron microscopy studies. The authors would have to measure the yield of exosomes, proteins and contaminants from both methods in parallel.

There are numerous publications presenting data comparing exosomal isolation methods that support the use of ultracentrifugation versus polymer-based precipitating methods such as ExoQuick. In particular, three papers demonstrate that up to 23 times more material is precipitated with ExoQuick compared to standard serial UC (Van Deun J et al., Journal of Extracellular Vesicles, 2014, 18(3), PMID: 25317274; Zlotogorski-Hurvitz A et al., J Histochem Cytochem. 2015, 63(3): 181-9, PMID: 25473095; Yamada T et al., J Vet Med Sci. 2012, 74(11):1523-5, PMID: 22785357). A recent study provides electron microscopy pictures that demonstrate that the higher protein yield obtained with the ExoQuick precipitation method compared to the UC gold standard is due to contaminants such as large protein aggregates (Sáenz-Cuesta M et al., Front Immunol, 2015 Feb 13;6:50; PMID: 25762995).

We have included a comparison of the two methods, ultracentrifugation and ExoQuick, as a pilot experiment (protocol 3) to occur prior to the in vivo experiment. This will compare protein yield, exosome number, and contaminants. This will allow for an adjustment, if needed, in the amount of isolated protein used in the education study to control for any potential differences in the number of exosomes relative to protein content isolated by each method. This will allow for the proposed education experiment to use an equivalent number of exosomes from 5 µg of total protein when generated by ultracentrifugation.

2) Importantly, the fact that the first three authors on the proposed Reproducibility project study are employees of System Biosciences, the company that produces ExoQuick as well as ExoFBS raises a red flag and creates a huge conflict on interest in terms of the choice of methodology. The choice of experimental approaches and reagents should be driven by the desire to perform the tests in conditions as close as possible to the original study, but the authors are clearly choosing a critical reagent that introduces a large variation from the original study based on financial interests.

To address the concern regarding the experimental approach, we have included a comparison between methods to the revised manuscript as described in point 1 above. Additionally, any known differences are stated a priori. We have also included in the conflict of interest statement that System Biosciences produces the two reagents in question as well as the TransDux and the CD63 antibody.3) It is not clear whether the authors plan to use fresh or frozen (-20°C) exosomal preparations for education. There have been studies that have shown that freezing exosomes results in lysis of more than 50% of the preparation (Oosthuyzen et al., J Physiol. 2013,591(Pt 23): 5833-5842) which will affect the amount delivered in a single dose (if the exosomes were quantified prior to freezing and that amount is used for calculation of material to inject for education). Freezing could also affect the functionality of exosomes. The original Peinado et al. study had performed the education studies with fresh exosomes.

The exosomes will be isolated, protein content determined, and used fresh for each injection. We have revised some of the language to attempt to make this clearer.

4) Another major problem is that the authors are using non-transduced B16-F10 cells as control for the B16-F10 shMET. It is highly recommended that a scrambled short hairpin control (shScramble) be used, as it is widely known that infection and puromycin treatment of cells may alter the cell population and the appropriate scrambled control in the same vector is available from the same company that produces the shMET. Importantly, the authors plan to rely solely on puromycin resistance to select the knockdown clones, but this is not sufficient to isolate a pure population of transduced cells that maintains high levels of vector integration and expression. In addition to verifying knockdown of MET, the authors also need to show that Met knockdown did not affect the growth of the cells (every integration event is unique and you never know where the lentivirus integrated).

We agree with this suggestion and have included the shScramble in the revised manuscript. It was not clear if it was used in Peinado et al. for Figure 4E where the cells were described as F10, not shScramble, unlike for the experiments reported in Figure 5 that describe the cells as shScramble.

The transduction efficiency (as monitored by GFP expression using fluorescent microscopy) will be recorded for each stable knockdown line and recorded. While sorting of the cells by FACS will not occur, the knockdown efficiency by western blot (protocol 2) will be compared to the original study to compare the achieved knockdown of the original and replication attempts. The revised manuscript includes additional details commenting on how detailed notes regarding cell passaging will be made available to assess if there are differences in cellular growth.

5) In protocol 1: The authors propose to use puromycin selection to generate stable B16-F10 shMet transfectants instead of FACS sorting GFP-positive cells, which should be sufficient. Upon reviewing the original paper, we could not find any details how the authors generated B16-F10 shMet cells, and whether these were stable transfectants. Was this communicated directly by the authors of the study?

Yes, this information was communicated directly by the authors of the original study. We have included additional text to highlight this.

6) It looks like they never tested whether MET was downregulated in exosomes themselves in supplementary Figure 5a, just in the parent cells from which they purified the exosomes. This is planned for the B16-F10 control and shMET exosomes, which we think is good. We would also suggest including B16-F1 exosomes, as there was a comparison by Western blot for the B16-F10 to B16-F1 exosomes in Figure 4a, at least that would be something to compare across studies.

Thank you for the suggestion. We had originally thought of including the B16-F1 exosomes as well, but considering the B16-F1 cells are not utilized elsewhere in the manuscript, we have not included it. While it would further help show that Met, and pMet, is present in B16-F10 cells with lower levels in B16-F1 cells, the shMet condition serves as the proper control for this. Additionally, since the B16-F1 and B16-F10 exosome comparison was not quantified in the original study, it is difficult to ascertain the relative level of Met or pMet in either cell line.

7) In the original paper there is a microarray analysis where genes with a fold change greater than 2 are reported. Apparently p-values from a t-test were computed with a permutation approach but not reported. It would be interesting to repeat also this experiment reporting the statistical significant genes controlling the false discovery rate.

We agree the microarray experiment would be interesting to replicate, however these types of experiments are excluded from all articles. These exclusion criteria are outlined on the project page (https://osf.io/e81xl/wiki/studies) and in a Feature Article describing the project (http://elifesciences.org/content/3/e04333). We understand that the exclusion of certain experiments limits the scope of what can be analyzed by the project, but we are attempting to identify a balance of breadth of sampling for general inference with sensible investment of resources on replication projects to determine to what extent the included experiments are reproducible. As such, we will restrict our analysis to the experiments being replicated and will not include discussion of experiments not being replicated in this study.

8) In the statistical analyses plan of the report it is claimed that Fisher's LSD correction has been used. However the alpha error 0.025 reported is due to a Bonferroni's correction; indeed the Fisher's LSD correction used by the authors is not taking into account that multiple comparisons will be performed and it's good only for the calculation of the effect size (see Hayter, 1986).

The Bonferroni correction (making the alpha error 0.025) was to account for the two measurements (lung photon flux and bones photon flux) on the same animals since they will be performed using two different test families (ANOVA and Kruskal-Wallis). However, within each measurement and test family alpha error of 0.025, the comparisons use a Fisher’s LSD correction. We agree with the reviewers comment on the use of a correction, such as Bonferroni or the modification of LSD by Hayter, as ways to control for the MFWER, however as Hayter describes in his 1986 paper, this applies in situations where the ANOVA is unbalanced or with a balanced design with four or more populations. Since the proposed analysis is balanced with three population groups, the LSD is sufficiently conservative and powerful to account for the multiple comparisons in this specific situation. This is further explained by Levin et al., 1994 and discussed in Maxwell and Delaney, 2004 (Chapter 5) and Cohen, 2001 (Chapter 12).

References:

Levin, J.R., Serline, R.C., & Seaman M.A. (1994). A controlled, powerful multiple-comparison strategy for several situations. Psychological Bulletin, 115, 153-159.

Maxwell, S.E. & Delaney, H.D. (2004). Designing experiments and analyzing data: a model comparison perspecitive. Lawrence Erlbaum Associates, Mahwah, N.J., 2nd edition.

Cohen, B.H. (2001). Explaining psychological statistics. John Wiley and Sons, New York, 2nd edition.

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Essential revisions:

1) We appreciate the authors' efforts to address my concerns. However, the new experiment (protocol 3) added to the manuscript proposing to compare ExoQuick extraction with the sequential ultracentrifugation presented in the original publication, is not appropriate for the "Reproducibility project". This is a new experiment that can be the focus of a new manuscript. Even for such a method comparison manuscript, to indeed validate their side-by-side comparison of ExoQuick and ultracentrifugation, the authors should perform unbiased analyses, specifically proteomic analysis of exosomes by mass spectrometry, not just western blot analyses. For the purpose of reproducing the data in the original Peinado et al. publication, the original exosome purification method should be used. Despite the fact that the differences from the original study are clearly stated for each protocol, for Protocols 2 and 4, the authors are still relying ONLY on ExoQuick for exosome purification. Given that during exosome isolation using ExoQuick, contaminants (e.g. VEGF) are trapped by ExoQuick, and the fact that education would not be strictly and solely due to exosomes, we believe that all the experiments proposed in this paper should be performed using the sequential ultracentrifugation, not ExoQuick. Moreover, the authors are using System Biosciences ExoFBS for all the experiments, rather than depleting exosomes from the FBS using sequential ultracentrifugation.

The revised manuscript includes the approach to purify exosomes by sequential ultracentrifugation instead of ExoQuick and as well as use FBS depleted of exosomes by ultracentrifugation as described in the original paper and confirmed by the original authors. Additionally, the protocol to compare ExoQuick to ultracentrifugation has been removed since it is no longer necessary for this replication attempt.

2) The authors declared their conflict of interest as employees of System Biosciences, the company that produces ExoQuick as well as ExoFBS. However, the determination of the authors to use System Biosciences reagents to conduct this study (even in the context of comparing methods) is impacted by the conflict of interest and driven by their desire to promote System Biosciences products. Therefore, because of their vested interest, employees of SBI should be excluded from the author list of this reproducibility project or the manuscript should not use any System Biosciences products.

We hope the reviewers agree that with the above revisions the conflict of interest has been adequately resolved. There are still two reagents used in the study that are produced by System Biosciences, TransDux, which replaces polybrene during the transfection procedure, and the rabbit anti-CD63 antibody, which was added as an additional quality control measure not included in the original study. Both are listed in the competing interests statement. As a reminder, TransDux reagent is similar to polybrene and enables high transduction rates of virus into most cells, sometimes at higher rates than polybrene. Most importantly, the knockdown efficiency, which is partly reflected by the infection reagent used, will be reported for both cells and exosomes, and for at the least the cells, will be compared to the originally reported knockdown efficiency.

3) The authors included the scrambled short hairpin control (shScramble) B16-F10 cells as s control for the B16-F10 shMET experiments. However, the authors plan to rely solely on puromycin resistance to select the knockdown clones, but this is not sufficient to isolate a pure population of transduced cells that maintains high levels of vector integration and expression. The authors should evaluate the knockdown after 28 days of puromycin selection. The authors should either use fluorescence-activated cell sorting to select a pure population or they should maintain the cells under puromycin selection. Just as another control, it would be desirable to compare both B16-F10 wild-type and Scramble in order to verify that the scramble does not have any undesirable effects, since infection can cause genome instability that way would be a perfect control.

The revised manuscript has the stable cells maintained in puromycin for at least 28 days prior to performing the knockdown evaluation. Additionally, the cells will remain under puromycin selection throughout maintenance of the cell lines, however when the cells are plated for experiments (for exosome purification) they will not include puromycin, which is similar to how exosomes were isolated in the original study.

We agree the addition of untransduced B16-F10 cells would be a valuable additional control to include. However, it is not feasible to include this extension to the original work since it is not possible to balance this additional aspect with the main aim of this project: to perform a direct replication of the original experiment(s).

4) The synthetic liposome control was a bit confusing. The authors say they will use 5 µg to match the protein on exosomes, but there should not be any protein on synthetic liposomes. We think they should figure out a vesicle concentration based on NanoSight and then use the same concentration of synthetic liposomes.

Thank you for highlighting this detail. In communication with the original authors we obtained the original amount of synthetic liposome controls injected and have added it to the revised manuscript. The amount of liposomes to inject is based on a theoretical ratio of 4:1 protein:L-α-phosphatidylcholine. With a formulation of liposomes that is 13 mg/ml L-α-phosphatidylcholine, 2.78 mg/ml cholesterol (7:3 molar ratio) the control injections will use 1.25 µg L-α-phosphatidylcholine to mimic the 5 µg of exosomes injected.

https://doi.org/10.7554/eLife.07383.003

Article and author information

Author details

  1. Jake Lesnik

    System Biosciences, LLC, Mountain View, United States
    Contribution
    JL, Drafting or revising the article
    Competing interests
    JL: System Biosciences Inc., is a Science Exchange associated lab. System Biosciences produces some of the reagents used in this study, specifically TransDux, and the rabbit anti-CD63 antibody.
  2. Travis Antes

    System Biosciences, LLC, Mountain View, United States
    Contribution
    TA, Drafting or revising the article
    Competing interests
    TA: System Biosciences Inc., is a Science Exchange associated lab. System Biosciences produces some of the reagents used in this study, specifically TransDux, and the rabbit anti-CD63 antibody.
  3. Jeewon Kim

    Transgenic Research Center, Stanford University School of Medicine, Stanford, United States
    Contribution
    JK, Transgenic Research Center is a Science Exchange associated lab., Drafting or revising the article
    Competing interests
    JK: This is a Science Exchange associated lab.
  4. Erin Griner

    University of Virginia, Charlottesville, United States
    Contribution
    EG, Drafting or revising the article
    Competing interests
    No competing interests declared.
  5. Luisa Pedro

    University of Cambridge, Cambridge, United Kingdom
    Contribution
    LP, Drafting or revising the article
    Competing interests
    No competing interests declared.
  6. Reproducibility Project: Cancer Biology

    Contribution
    RP:CB, Conception and design, Drafting or revising the article
    For correspondence
    tim@cos.io
    Competing interests
    No competing interests declared.
    1. Elizabeth Iorns, Science Exchange, Palo Alto, United State
    2. William Gunn, Mendeley, London, United Kingdom
    3. Fraser Tan, Science Exchange, Palo Alto, United States
    4. Joelle Lomax, Science Exchange, Palo Alto, United States
    5. Nicole Perfito, Science Exchange, Palo Alto, United States
    6. Timothy Errington, Center for Open Science, Charlottesville, United States

Funding

Laura and John Arnold Foundation

  • Reproducibility Project: Cancer Biology

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

The Reproducibility Project: Cancer Biology core team would like to thank the original authors, in particular Hector Peinado, for generously sharing critical information as well as reagents to ensure the fidelity and quality of this replication attempt. We thank Courtney Soderberg at the Center for Open Science for assistance with statistical analyses. We thank Maureen Peterson at System Biosciences for help with some protocol details. We would also like to thank the following companies for generously donating reagents to the Reproducibility Project: Cancer Biology: American Type Culture Collection (ATCC), Applied Biological Materials, BioLegend, Charles River Laboratories, Corning Incorporated, DDC Medical, EMD Millipore, Harlan Laboratories, LI-COR Biosciences, Mirus Bio, Novus Biologicals, Sigma-Aldrich, and System Biosciences (SBI).

Reviewing Editor

  1. Michael R Green, Reviewing Editor, Howard Hughes Medical Institute, University of Massachusetts Medical School, United States

Publication history

  1. Received: March 9, 2015
  2. Accepted: October 28, 2015
  3. Version of Record published: January 29, 2016 (version 1)

Copyright

© 2016, Lesnik et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 1,464
    Page views
  • 349
    Downloads
  • 2
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, Scopus, PubMed Central.

Comments

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)

Further reading

  1. H Peinado et al.
    1
    1. Cancer Biology
    Curated by Roger Davis et al.
    Collection Updated

    Investigating reproducibility in preclinical cancer research.

    1. Cell Biology
    2. Human Biology and Medicine
    Xunde Xian et al.
    Research Article