Understanding the genetic basis of evolution requires the identification of genes and mutations responsible for phenotypic variation between individuals and populations. A complete catalog of such variants will allow the identification of the genetic paths favored by evolution (Stern and Orgogozo, 2008; Martin and Orgogozo, 2013). Regarding morphological variation, the latest catalog, that includes animals, yeasts, and plants, listed 386 alleles (Martin and Orgogozo, 2013), most of them related to pigmentation. Genes associated with shape variation are fewer, and correspond to variation in wing shape in flies and butterflies, and body shape in fish. However, there have been many other studies exploring the genetic basis of shape variation (e.g. craniofacial shape), but given the highly polygenic nature of these traits it has been difficult to validate the extensive list of candidate genes identified through quantitative trait loci (QTL) and genome-wide association (GWAS) approaches.

The mouse tooth is one of the traits extensively studied regarding shape variation. In paleontology, it is a key character for phylogenetic and dietary inferences (Misonne, 1969; Gómez Cano et al., 2013). It has been a model for developmental genetics (Jernvall and Thesleff, 2012; Urdy et al., 2016). The genes and pathways involved in tooth morphogenesis are well known (Thesleff and Sharpe, 1997; Jernvall and Thesleff, 2000; Jernvall and Thesleff, 2012; Lan et al., 2014) and a few genes affecting cusp patterning in mice have been identified (Jernvall and Thesleff, 2012). Moreover, computational models have been able to recreate the morphological transition between species and individuals (Salazar-Ciudad and Jernvall, 2002, 2010; Harjunmaa et al., 2014; Urdy et al., 2016). As for many other traits, there is a wide gap between the macro- and micro-evolutionary understanding of tooth morphological variation (Nunes et al., 2013). The wide-spread morphological variation at the micro-evolutionary level in mice (i.e. within-species) has been repeatedly highlighted (Boell and Tautz, 2011; Ledevin et al., 2016), as well as its relevance for understanding the evolutionary history of the taxon (Macholán, 2006; Renaud et al., 2011; Renaud and Auffray, 2013; Renaud et al., 2013).

Surprisingly, only two studies have tried to map loci responsible for such morphological variation. Using F2 crosses between LG/J and SM/J inbred lines of mice, Workman et al., 2002 found three QTL for molar row size, and 18 QTL for 2D molar row shape. Using recombinant inbred lines between A/J and SM/J, Shimizu et al. (2004) found seven QTL affecting size of either the upper or lower molars. As is common for QTL approaches, hundreds of genes co-localized with the QTL, impeding the identification of clear candidate loci.

In this study, we focused on the first upper molar and used mice derived from a wild population, which allowed us to directly address the genetic basis of molar shape variation from a micro-evolutionary perspective. The mice used here are first-generation offspring of natural hybrids between Mus musculus musculus and Mus musculus domesticus collected in the Bavarian hybrid zone (Turner et al., 2012). As a result, phenotypic variation in the sample has a between-subspecies as well as a within-population component. This sample represents phenotypic and genetic variation segregating in nature, and therefore constitutes an evolutionarily relevant scenario in contrast to inbred lines traditionally used in mapping studies in mice. Previous studies successfully identified loci associated with cranial morphology and male sterility phenotypes in this mapping population (Pallares et al., 2014; Turner and Harr, 2014). We performed the first GWAS of 3D molar shape variation, and identified several candidate regions for naturally occurring variation. We also, for the first time, showed that mutations in the candidate gene Mitf can cause between-individual differences in molar shape in the mouse.

The mice used in this study were derived from wild-caught hybrid mice between M. m. musculus and M. m. domesticus and represent a hybridization continuum between the two subspecies (Figure 1, a). Molars of M. m. musculus are characterized by anterior elongation, expansion of the labial cusps, and reduction of the antero-lingual cusp compared to M. m. domesticus mice (Figure 1, b). Despite the hybrid character of the sample, the major axes of variation are not polarized by shape differences between the two subspecies (Figure 1, c). This indicates that phenotypic variation in the sample has a between-subspecies component, but axes of within-subspecies variation are more important, representing other directions of shape changes different from M. m. musculus – M. m. domesticus. This suggests that within-population variation in molar shape is larger than between-subspecies variation, and therefore species-specific alleles might not be playing any major role in tooth shape in this population. A similar pattern of between- vs within-population variation has been previously described for mandible shape (Boell and Tautz, 2011). Additional factors that might contribute to such a pattern are transgressive phenotypes and hybrid developmental instability, although the latter seems to be not very important in this population (see Pallares et al., 2016).

Figure 1

Download asset Open asset

Tooth shape, mouse age and wear

Once erupted, tooth shape does not change except by the effect of wear, but the degree of wear is correlated with the age of the mice, that in this study ranges from 9 to 12 weeks. The effect of age and wear on molar tooth shape was explored using the first 18 PCs derived from each of the two approaches: complete template and wear-free template. Age differences have a small but significant effect on molar shape variation when using the complete template (p-value=5.4×10⁻⁵, r²=0.01), and this is reflected in the significant correlation between age and some PCs (r²(PC1) = 2.1%, p-value=0.027; r²(PC4) = 2.2%, p-value=0.025; r²(PC5) = 3.3%, p-value=0.008; r²(PC14) = 1.8%, p-value=0.04; r²(PC18) = 4.9%, p-value=0.002). More importantly, a qualitative assessment indicates that wear-like patterns are present already in the first axis of variation (PC1) (Figure 2-a).

Figure 2

Download asset Open asset

Although age correlates with wear patterns, there are other factors, such as diet and behavior that also influence tooth morphology through wear (Renaud and Ledevin, 2017). The wear-free template lacks the tip of the cusps, mimicking the same level of wear in all individuals (Figure 2-b), and therefore addressing wear in all its complexity, being this related to age or any other factors. The effect of age on overall shape variation using this template was still significant but explained very little variance (p-value=0.01, r²=0.01). Only two PCs kept a very small age-related signal (r²(PC3) = 4.6%, p-value=0.002; r²(PC18) = 3.5, p-value=0.007).

Heritability

The heritability values estimated in this study correspond to the effect that all SNPs used in the mapping have on the phenotype; this value is also known as SNP or chip heritability, and it serves as a proxy of the additive genetic variance underlying phenotypic variation. Substantial genetic variance was found in all PCs (Table 2). A weighted average of all PCs genetic variance was used to summarize the total heritability of molar shape (see Materials and methods). This resulted in a value of 65.5%, indicating that more than half of shape variation is accounted for by additive genetic effects.

Table 2

PC	%var	Heritability per PC	Error	Molar herit
1	18.9	0.83	0.09	15.8
2	15.4	0.95	0.08	14.7
3	9.9	0.78	0.10	7.7
4	7.1	0.62	0.14	4.4
5	5.9	0.49	0.12	2.9
6	5.1	0.83	0.10	4.2
7	3.9	0.53	0.12	2.1
8	3.2	0.89	0.11	2.8
9	2.8	0.89	0.09	2.5
10	2.5	0.86	0.13	2.2
11	2.1	0.72	0.14	1.5
12	1.8	0.60	0.15	1.1
13	1.7	0.68	0.13	1.2
14	1.5	0.56	0.16	0.8
15	1.4	0.53	0.15	0.7
16	1.3	0.26	0.16	0.3
17	1.2	0.14	0.20	0.2
18	1	0.51	0.14	0.5
Total Var	86.7%
		pve for molar shape		65.50

When exploring heritability from a chromosomal point of view (Figure 3), instead of individual associations between SNPs and phenotype, a positive correlation is evident between the amount of phenotypic variation explained by each chromosome and chromosomal length (r = 0.67, p-value=0.001). This is the expected pattern when the effect of individual SNPs is small, and such SNPs are many and homogeneously distributed along the genome.

Figure 3

Download asset Open asset

Mapping of loci associated with molar shape variation

We decomposed variation in molar tooth shape in PCs. The mapping was performed using 183 mice and ~145,000 SNPs. Centroid size was used as proxy for tooth size, however, no genomic regions were significantly associated with size variation.

When shape data derived from the complete template was used in the mapping, no significant associations were found. In contrast, the data derived from the wear-free template resulted in nine SNPs significantly associated with molar shape variation, clustered in five genomic regions. Hereafter, we will thus focus on the results of the wear-free approach.

These five loci are found in chromosomes 1, 5, 6, and X, and were associated with PC7, PC11, PC16, and PC18 (Figure 4-a, Table 3). The name of each locus is defined by Mo (molar) and chromosomal location. Regions Mo.1, Mo.5, Mo.6, and Mo.X.1 have an arbitrary size of 500 Kb (see Materials and methods) and contain, together, 15 protein coding genes. Mo.X.2 is 63.8 Mb given the strong LD pattern around the best SNP and contains 306 protein coding genes. Together, the loci explain ~10% of molar shape variation, with individual effects ranging from 1% to 3% (Table 3). The shape changes associated with each region were estimated as the difference between the consensus shape of the two homozygous states for each SNP. The phenotypic effect of each SNP is therefore not restricted to the PC it was associated with, but represents the effect of the SNP on the complete shape dimensionality (Figure 4-b).

Figure 4

Download asset Open asset

Figure 4—source data 1

Shape data used in the association mapping.

The dimensionality of the molar shape data was reduced using a PCA. The centroid size and PC scores for the first 18 PCs for each mouse are shown. These PC scores were used as phenotypes in the association mapping implemented in GEMMAX.

https://doi.org/10.7554/eLife.29510.008

Download elife-29510-fig4-data1-v2.txt

Table 3

QTL	Chr	Position	Best SNP	p-value	Effect size	PC axis	Genes*
Mo.1	chr1	84306638	JAX00006079	5.11E-07	1.1%	PC11	Pid1, Dner
Mo.5	chr5	36723779	JAX00128837	5.79E-07	3.2%	PC18	Psapl1, Tada2b, Ccdc96, Grpel1, Tbc1d14, D5Ertd579e, Sorcs2
Mo.6	chr6	97980057	JAX00619074	2.71E-08	2.8%	PC7	Gm765, Mitf
Mo.X.1	chrX	92638616	JAX00715960	1.18E-07	1.6%	PC16	Fam123b, Zc4h2, Asb12, Arhgef9
Mo.X.2**	chrX	104533418	JAX00183055	1.28E-10	2.2%	PC7	Rps6ka3, Dach2, Ap1s2, Itm2a and 301 other genes

Mitf gene affects molar shape in mice

From the genes located in the QTL regions, two have been reported to affect tooth development directly or indirectly (MGI database queried 22.03.16), namely Rps6ka3 (Laugel-Haushalter et al., 2014) and the microphthalmia-associated transcription factor Mitf (Al-Douri and Johnson, 1987). The Rps6ka3 gene is located in Mo.X.2 that contains 305 additional genes, making it difficult to assess its relevance. However, Mitf is one of two genes found in Mo.6, and since many different mutations are known in this gene, it is possible to determine its relevance in tooth development.

Mitf is mainly known for its role in melanocyte development and proliferation in mice and humans. Mice carrying Mitf mutations show reduced or absent pigmentation, reduced eye size (microphthalmia), and deafness (Steingrimsson et al., 2004). Some mutations result in defective bone resorption due to defects in osteoclast development (Hodgkinson et al., 1993; Moore, 1995; Hershey and Fisher, 2004); the Mitf^mi allele leads to lethality at 3 weeks of age due to severe osteopetrosis. Abnormal morphology and eruption of incisor and molar teeth has been reported in some mutants (Al-Douri and Johnson, 1987; Steingrimsson et al., 2002), although this is thought to be linked to severe osteopetrosis in the mandible of these mutants.

Interestingly, the large Mo.X.2 region, which is associated with the same PC as Mitf, includes Ap1s2. This gene is a target of the Mitf-regulated miRNA miR-211, and its function has been validated in the context of melanoma (Margue et al., 2013) and osteosclerosis in humans (Saillour et al., 2007).

To further evaluate a possible role of Mitf in molar shape variation, we examined molar morphology in mice carrying four mutant Mitf alleles. To avoid indirect effects on molar morphogenesis, we selected alleles for which no, or very mild osteopetrosis has been described (Table 1). Mitf^mi induces severe osteopetrosis when homozygous (Steingrimsson et al., 2002). However, we have only used it in the heterozygous state. The other mutant alleles do not exhibit osteopetrosis in homozygotes (Steingrimsson et al., 2002; Steingrimsson et al., 2004).

As shown in Figure 5, all Mitf alleles affected the shape of the upper first molar. Pairwise tests between wild type B6 mice and each mutant group showed significant differences in mean shape (Supplementary file 1A). In terms of Procrustes distances, Mitf^{mi-enu22(398)} is closest to wild type (0.0237), and Mitf^mi-vga9/Mitfmi^mi-vga9 is most distant (0.0466) (Supplementary file 1A).

Figure 5

Download asset Open asset

Mitf^Mi-wh/Mitf^mi compound heterozygotes have a similar molar shape as Mitf^Mi-wh/+ heterozygotes, suggesting that the Mitf^mi allele alone has no additional effect on molar shape (Figure 5d,g). Interestingly, in homozygous condition, Mitf^mi does result in severe pigmentation and eye phenotypes (Steingrimsson et al., 2004). However, given that Mitf^mi/+ mice were not examined in this study, we are not able to rule out the role of this allele in tooth shape. The molar shape generated by Mitf^mi-vga9/+ is very similar to Mitf^Mi-wh/Mitf^Mi-wh, although the intensity of the effect seems stronger in the latter (Figure 5b,e). For the Mitf^mi-vga9 and Mitf^Mi-wh alleles, three genetic combinations were available. The Mitf^mi-vga9 mutation behaves additively regarding molar shape (Figure 5). This contrasts with reports on other phenotypes where its effect is clearly recessive (Hodgkinson et al., 1993).

A comparison between each mutant group and Mo.6, the region where Mitf gene is found, shows that mice homozygous for the Mitf^Mi-wh allele are most similar to Mo.6 with respect to tooth shape. Mitf^mi-vga9/Mitfmi^mi-vga9 follows second in correlation strength (Figure 5h). The magnitude of shape variation generated by the mutant alleles falls within the range of shape variation observed in the mapping population (Figure 5i). However, it should be noted that the shape variation represented by the hybrid mice used here is determined by many genes, while the shape of mutant mice is the result of a mutation in a single gene.

Using genome-wide association mapping and 3D surface morphometrics we were able to map genomic regions underlying shape variation in the first upper molar of wild mice. The high mapping resolution achieved enabled identification of individual candidate genes making it feasible to functionally evaluate such candidates. Using a panel of mice with different mutant genotypes, we showed that one of these candidates, the transcription factor Mitf, has significant effects on molar shape.

The mapping population and the phenotyping approach used here played a definitive role in the ability to identify the genetic basis of molar shape variation. The mice were derived from wild-caught parents collected in the Bavarian hybrid zone in Germany (Turner et al., 2012) and therefore are representative of wild genomic and phenotypic variation. The phenotypic changes driving the first axis of variation do not correspond to the shape changes between M.m.musculus and M.m.domesticus, indicating that this sample contains not only between-species patterns of variation but also strong within-subspecies variation. Such variation can be the result of transgressive segregation or developmental instability, often associated with hybridization. However, the latter seem to play a very small role in this population (Pallares et al., 2016). This is the first time the genetics of molar shape variation in mice has been studied using wild mice, offering an evolutionarily relevant perspective for the understanding of phenotypic variation. The technical advantage comes from the fact that these mice are hybrids between M.m.musculus and M.m.domesticus. These subspecies have been hybridizing in the Bavarian region for around 3000 years (reviewed in Baird and Macholán, 2012) resulting in high mapping resolution as a consequence of a genomic landscape where LD blocks are much smaller compared to traditional QTL-mapping approaches which usually correspond to two generations of crossing (Workman et al., 2002). The second technical advantage comes from the reduction of environmental variation; the mice were raised in the laboratory under controlled conditions and therefore the relative effect of genetic variation is enhanced at the expense of environmental variation. The suitability of this population for mapping loci associated with complex traits has been demonstrated previously for craniofacial shape (Pallares et al., 2014) and sterility phenotypes (Turner and Harr, 2014).

The phenotyping approach used here made use of 3D surface morphometrics, allowing us to quantify additional dimensions of variation compared to the two-dimensional approach (Workman et al., 2002; Shimizu et al., 2004). By measuring the surface of the tooth, this approach captures variation generated by differential wear between individuals, a confounding factor that is not present in 2D studies. Following Ledevin et al. (2016), we used a wear-free template that allowed us to preserve the additional shape information captured by 3D methods and exclude wear-related variation. In this way, we were able to identify genetic loci associated with shape variation that otherwise would have been obscured by strong wear effects (see Results). Our findings are, however, a subset of the possible associations between molar shape and genomic regions. Between-individual differences in cusp shape remain to be explored at the genomic level.

1. 1. Pallares LF
   2. Turner LM
   3. Harr B
   4. Tautz D

   (2014) Use of a natural hybrid zone for genome-wide association mapping of craniofacial traits in the house mouse

   Available at Dryad Digital Repository under a CC0 Public Domain Dedication.

   http://dx.doi.org/10.5061/dryad.bt848

1. 1. Al-Douri SM
   2. Johnson DR

   (1987)

   The development of the teeth of the microphthalmic (mi/mi) mouse

   Journal of Anatomy 153:139-49.

   • PubMed
   • Google Scholar
2. 1. Baird SJE
   2. Macholán M

   (2012) What can the Mus musculus musculus/M. m. domesticus hybrid zone tell us about speciation?

   Evolution of the House Mouse pp. 334–372.

   https://doi.org/10.1017/CBO9781139044547.016

   • Google Scholar
3. 1. Bauer GL
   2. Praetorius C
   3. Bergsteinsdóttir K
   4. Hallsson JH
   5. Gísladóttir BK
   6. Schepsky A
   7. Swing DA
   8. O'Sullivan TN
   9. Arnheiter H
   10. Bismuth K
   11. Debbache J
   12. Fletcher C
   13. Warming S
   14. Copeland NG
   15. Jenkins NA
   16. Steingrímsson E

   (2009) The role of MITF phosphorylation sites during coat color and eye development in mice analyzed by bacterial artificial chromosome transgene rescue

   Genetics 183:581–594.

   https://doi.org/10.1534/genetics.109.103945

   • PubMed
   • Google Scholar
4. 1. Bei M

   (2009) Molecular genetics of tooth development

   Current Opinion in Genetics & Development 19:504–510.

   https://doi.org/10.1016/j.gde.2009.09.002

   • PubMed
   • Google Scholar
5. 1. Boell L
   2. Tautz D

   (2011) Micro-evolutionary divergence patterns of mandible shapes in wild house mouse (Mus musculus) populations

   BMC Evolutionary Biology 11:306.

   https://doi.org/10.1186/1471-2148-11-306

   • PubMed
   • Google Scholar
6. 1. Bookstein FL

   (1997) Landmark methods for forms without landmarks: morphometrics of group differences in outline shape

   Medical Image Analysis 1:225–243.

   https://doi.org/10.1016/S1361-8415(97)85012-8

   • PubMed
   • Google Scholar
7. Book

   1. Gunz P
   2. Mitteroecker P

   (2005)

   Semilandmarks in Three Dimensions

   In: Slice D. E, editors. Modern Morphometrics in Physical Anthropology. Boston: Springer. pp. 73–98.

   • Google Scholar
8. 1. Gómez Cano AR
   2. Hernández Fernández M
   3. Alvarez-Sierra MÁ

   (2013) Dietary ecology of Murinae (Muridae, Rodentia): a geometric morphometric approach

   PLoS ONE 8:e79080.

   https://doi.org/10.1371/journal.pone.0079080

   • PubMed
   • Google Scholar
9. 1. Harjunmaa E
   2. Seidel K
   3. Häkkinen T
   4. Renvoisé E
   5. Corfe IJ
   6. Kallonen A
   7. Zhang ZQ
   8. Evans AR
   9. Mikkola ML
   10. Salazar-Ciudad I
   11. Klein OD
   12. Jernvall J

   (2014) Replaying evolutionary transitions from the dental fossil record

   Nature 512:44–48.

   https://doi.org/10.1038/nature13613

   • PubMed
   • Google Scholar
10. 1. Harr B
    2. Karakoc E
    3. Neme R
    4. Teschke M
    5. Pfeifle C
    6. Pezer Ž
    7. Babiker H
    8. Linnenbrink M
    9. Montero I
    10. Scavetta R
    11. Abai MR
    12. Molins MP
    13. Schlegel M
    14. Ulrich RG
    15. Altmüller J
    16. Franitza M
    17. Büntge A
    18. Künzel S
    19. Tautz D

    (2016) Genomic resources for wild populations of the house mouse, Mus musculus and its close relative Mus spretus

    Scientific Data 3:75.

    https://doi.org/10.1038/sdata.2016.75

    • PubMed
    • Google Scholar
11. 1. Hershey CL
    2. Fisher DE

    (2004) Mitf and Tfe3: members of a b-HLH-ZIP transcription factor family essential for osteoclast development and function

    Bone 34:689–696.

    https://doi.org/10.1016/j.bone.2003.08.014

    • PubMed
    • Google Scholar
12. 1. Hodgkinson CA
    2. Moore KJ
    3. Nakayama A
    4. Steingrímsson E
    5. Copeland NG
    6. Jenkins NA
    7. Arnheiter H

    (1993) Mutations at the mouse microphthalmia locus are associated with defects in a gene encoding a novel basic-helix-loop-helix-zipper protein

    Cell 74:395–404.

    https://doi.org/10.1016/0092-8674(93)90429-T

    • PubMed
    • Google Scholar
13. 1. Jernvall J
    2. Thesleff I

    (2000) Reiterative signaling and patterning during mammalian tooth morphogenesis

    Mechanisms of Development 92:19–29.

    https://doi.org/10.1016/S0925-4773(99)00322-6

    • PubMed
    • Google Scholar
14. 1. Jernvall J
    2. Thesleff I

    (2012) Tooth shape formation and tooth renewal: evolving with the same signals

    Development 139:3487–3497.

    https://doi.org/10.1242/dev.085084

    • PubMed
    • Google Scholar
15. 1. Kavanagh KD
    2. Evans AR
    3. Jernvall J

    (2007) Predicting evolutionary patterns of mammalian teeth from development

    Nature 449:427–432.

    https://doi.org/10.1038/nature06153

    • PubMed
    • Google Scholar
16. 1. Lan Y
    2. Jia S
    3. Jiang R

    (2014) Molecular patterning of the mammalian dentition

    Seminars in Cell & Developmental Biology 25-26:61–70.

    https://doi.org/10.1016/j.semcdb.2013.12.003

    • PubMed
    • Google Scholar
17. 1. Laugel-Haushalter V
    2. Paschaki M
    3. Marangoni P
    4. Pilgram C
    5. Langer A
    6. Kuntz T
    7. Demassue J
    8. Morkmued S
    9. Choquet P
    10. Constantinesco A
    11. Bornert F
    12. Schmittbuhl M
    13. Pannetier S
    14. Viriot L
    15. Hanauer A
    16. Dollé P
    17. Bloch-Zupan A

    (2014) RSK2 is a modulator of craniofacial development

    PLoS ONE 9:e84343.

    https://doi.org/10.1371/journal.pone.0084343

    • PubMed
    • Google Scholar
18. 1. Ledevin R
    2. Chevret P
    3. Ganem G
    4. Britton-Davidian J
    5. Hardouin EA
    6. Chapuis J-L
    7. Pisanu B
    8. da Luz Mathias M
    9. Schlager S
    10. Auffray J-C
    11. Renaud S

    (2016) Phylogeny and adaptation shape the teeth of insular mice

    Proceedings of the Royal Society B: Biological Sciences 283:20152820.

    https://doi.org/10.1098/rspb.2015.2820

    • Google Scholar
19. 1. Macholán M

    (2006) A geometric morphometric analysis of the shape of the first upper molar in mice of the genus Mus (Muridae, Rodentia)

    Journal of Zoology 270:672–681.

    https://doi.org/10.1111/j.1469-7998.2006.00156.x

    • Google Scholar
20. 1. Margue C
    2. Philippidou D
    3. Reinsbach SE
    4. Schmitt M
    5. Behrmann I
    6. Kreis S

    (2013) New target genes of MITF-induced microRNA-211 contribute to melanoma cell invasion

    PLoS ONE 8:e73473.

    https://doi.org/10.1371/journal.pone.0073473

    • PubMed
    • Google Scholar
21. 1. Martin A
    2. Orgogozo V

    (2013) The loci of repeated evolution: a catalog of genetic hotspots of phenotypic variation

    Evolution 22:1235–1250.

    https://doi.org/10.1111/evo.12081

    • Google Scholar
22. 1. Misonne X

    (1969)

    African and indo-australian muridae evolutionary trends

    Annales Du Musée Royal d’Afrique Centrale Tervuren 172:1–219.

    • Google Scholar
23. 1. Monteiro LR
    2. Diniz-Filho JA
    3. dos Reis SF
    4. Araújo ED
    5. Diniz JAF

    (2002)

    Geometric estimates of heritability in biological shape

    Evolution 56:563–572.

    • Google Scholar
24. 1. Monteiro LR
    2. Diniz-Filho JA
    3. dos Reis SF
    4. Araújo ED

    (2003)

    Shape distances in general linear models: are they really at odds with the goals of morphometrics? A reply to Klingenberg

    Evolution 57:196–199.

    • Google Scholar
25. 1. Moore KJ

    (1995) Insight into the microphthalmia gene

    Trends in Genetics 11:442–448.

    https://doi.org/10.1016/S0168-9525(00)89143-X

    • PubMed
    • Google Scholar
26. 1. Nunes MD
    2. Arif S
    3. Schlötterer C
    4. McGregor AP

    (2013) A perspective on micro-evo-devo: progress and potential

    Genetics 195:625–634.

    https://doi.org/10.1534/genetics.113.156463

    • PubMed
    • Google Scholar
27. 1. Pallares LF
    2. Harr B
    3. Turner LM
    4. Tautz D

    (2014) Use of a natural hybrid zone for genomewide association mapping of craniofacial traits in the house mouse

    Molecular Ecology 23:5756–5770.

    https://doi.org/10.1111/mec.12968

    • PubMed
    • Google Scholar
28. 1. Pallares LF
    2. Harr B

    (2014)

    Data from: use of a natural hybrid zone for genome-wide association mapping of craniofacial traits in the house mouse

    Dryad Data Repository.

    • Google Scholar
29. 1. Pallares LF
    2. Carbonetto P
    3. Gopalakrishnan S
    4. Parker CC
    5. Ackert-Bicknell CL
    6. Palmer AA
    7. Tautz D

    (2015) Mapping of craniofacial traits in outbred mice identifies major developmental genes involved in shape determination

    PLoS Genetics 11:e1005607.

    https://doi.org/10.1371/journal.pgen.1005607

    • PubMed
    • Google Scholar
30. 1. Pallares LF
    2. Turner LM
    3. Tautz D

    (2016) Craniofacial shape transition across the house mouse hybrid zone: implications for the genetic architecture and evolution of between-species differences

    Development Genes and Evolution 226:173–186.

    https://doi.org/10.1007/s00427-016-0550-7

    • PubMed
    • Google Scholar
31. 1. Purcell S
    2. Neale B
    3. Todd-Brown K
    4. Thomas L
    5. Ferreira MA
    6. Bender D
    7. Maller J
    8. Sklar P
    9. de Bakker PI
    10. Daly MJ
    11. Sham PC

    (2007) PLINK: a tool set for whole-genome association and population-based linkage analyses

    The American Journal of Human Genetics 81:559–575.

    https://doi.org/10.1086/519795

    • PubMed
    • Google Scholar
32. 1. Renaud S
    2. Pantalacci S
    3. Auffray JC

    (2011) Differential evolvability along lines of least resistance of upper and lower molars in island house mice

    PLoS ONE 6:e18951.

    https://doi.org/10.1371/journal.pone.0018951

    • PubMed
    • Google Scholar
33. 1. Renaud S
    2. Auffray JC

    (2013)

    The direction of main phenotypic variance as a channel to morphological evolution: case studies in murine rodents

    Hystrix : Rivista Di Teriologia 24:85–93.

    • Google Scholar
34. 1. Renaud S
    2. Hardouin EA
    3. Pisanu B
    4. Chapuis JL

    (2013) Invasive house mice facing a changing environment on the Sub-antarctic guillou island (kerguelen archipelago)

    Journal of Evolutionary Biology 26:612–624.

    https://doi.org/10.1111/jeb.12079

    • PubMed
    • Google Scholar
35. 1. Renaud S
    2. Hardouin EA
    3. Quéré J-P
    4. Chevret P

    (2017) Morphometric variations at an ecological scale: Seasonal and local variations in feral and commensal house mice

    Mammalian Biology - Zeitschrift Für Säugetierkunde 87:1–12.

    https://doi.org/10.1016/j.mambio.2017.04.004

    • Google Scholar
36. 1. Renaud S
    2. Ledevin R

    (2017) Impact of wear and diet on molar row geometry and topography in the house mouse

    Archives of Oral Biology 81:31–40.

    https://doi.org/10.1016/j.archoralbio.2017.04.028

    • PubMed
    • Google Scholar
37. 1. Rodrigues HG
    2. Renaud S
    3. Charles C
    4. Le Poul Y
    5. Solé F
    6. Aguilar JP
    7. Michaux J
    8. Tafforeau P
    9. Headon D
    10. Jernvall J
    11. Viriot L

    (2013) Roles of dental development and adaptation in rodent evolution

    Nature Communications 4:2504.

    https://doi.org/10.1038/ncomms3504

    • PubMed
    • Google Scholar
38. 1. Saillour Y
    2. Zanni G
    3. Des Portes V
    4. Heron D
    5. Guibaud L
    6. Iba-Zizen MT
    7. Pedespan JL
    8. Poirier K
    9. Castelnau L
    10. Julien C
    11. Franconnet C
    12. Bonthron D
    13. Porteous ME
    14. Chelly J
    15. Bienvenu T

    (2007) Mutations in the AP1S2 gene encoding the sigma 2 subunit of the adaptor protein 1 complex are associated with syndromic X-linked mental retardation with hydrocephalus and calcifications in basal ganglia

    Journal of Medical Genetics 44:739–744.

    https://doi.org/10.1136/jmg.2007.051334

    • PubMed
    • Google Scholar
39. 1. Salazar-Ciudad I
    2. Jernvall J

    (2002) A gene network model accounting for development and evolution of mammalian teeth

    PNAS 99:8116–8120.

    https://doi.org/10.1073/pnas.132069499

    • PubMed
    • Google Scholar
40. 1. Salazar-Ciudad I
    2. Jernvall J

    (2010) A computational model of teeth and the developmental origins of morphological variation

    Nature 464:583–586.

    https://doi.org/10.1038/nature08838

    • PubMed
    • Google Scholar
41. 1. Schlager S

    (2016) Morpho: calculations and visualizations related to geometric morphometrics

    Morpho: calculations and visualizations related to geometric morphometrics, R package version 2.3.1.1, https://CRAN.R-project.org/package=Morpho.

    https://CRAN.R-project.org/package=Morpho

    • Google Scholar
42. 1. Shimizu T
    2. Oikawa H
    3. Han J
    4. Kurose E
    5. Maeda T

    (2004) Genetic analysis of crown size in the first molars using SMXA recombinant inbred mouse strains

    Journal of Dental Research 83:45–49.

    https://doi.org/10.1177/154405910408300109

    • PubMed
    • Google Scholar
43. 1. Steingrimsson E
    2. Tessarollo L
    3. Pathak B
    4. Hou L
    5. Arnheiter H
    6. Copeland NG
    7. Jenkins NA

    (2002) Mitf and Tfe3, two members of the Mitf-Tfe family of bHLH-Zip transcription factors, have important but functionally redundant roles in osteoclast development

    PNAS 99:4477–4482.

    https://doi.org/10.1073/pnas.072071099

    • PubMed
    • Google Scholar
44. 1. Steingrimsson E
    2. Arnheiter H
    3. Hallsson JH
    4. Lamoreux ML
    5. Copeland NG
    6. Jenkins NA

    (2003)

    Interallelic complementation at the mouse Mitf locus

    Genetics 163:267–276.

    • PubMed
    • Google Scholar
45. 1. Steingrimsson E
    2. Copeland NG
    3. Jenkins NA

    (2004) Melanocytes and the microphthalmia transcription factor network

    Annual Review of Genetics 38:365–411.

    https://doi.org/10.1146/annurev.genet.38.072902.092717

    • PubMed
    • Google Scholar
46. 1. Stern DL
    2. Orgogozo V

    (2008) The loci of evolution: how predictable is genetic evolution?

    Evolution 62:2155–2177.

    https://doi.org/10.1111/j.1558-5646.2008.00450.x

    • PubMed
    • Google Scholar
47. 1. Thesleff I
    2. Sharpe P

    (1997) Signalling networks regulating dental development

    Mechanisms of Development 67:111–123.

    https://doi.org/10.1016/S0925-4773(97)00115-9

    • PubMed
    • Google Scholar
48. 1. Turner LM
    2. Schwahn DJ
    3. Harr B

    (2012) Reduced male fertility is common but highly variable in form and severity in a natural house mouse hybrid zone

    Evolution 66:443–458.

    https://doi.org/10.1111/j.1558-5646.2011.01445.x

    • Google Scholar
49. 1. Turner LM
    2. Harr B

    (2014) Genome-wide mapping in a house mouse hybrid zone reveals hybrid sterility loci and Dobzhansky-Muller interactions

    eLife 3:e02504.

    https://doi.org/10.7554/eLife.02504

    • Google Scholar
50. 1. Turner LM
    2. Harr B

    (2014a)

     Data from: Genome-wide mapping in a house mouse hybrid zone reveals hybrid sterility loci and Dobzhansky-Muller interactions

     Dryad Data Repository.

    • Google Scholar
51. 1. Urdy S
    2. Goudemand N
    3. Pantalacci S

    (2016) Looking beyond the genes: the interplay between signaling pathways and mechanics in the shaping and diversification of epithelial tissues

    Current Topics in Developmental Biology 119:227–290.

    https://doi.org/10.1016/bs.ctdb.2016.03.005

    • PubMed
    • Google Scholar
52. 1. Valenzuela-Lamas S
    2. Baylac M
    3. Cucchi  T
    4. Vigne J-D

    (2011) House mouse dispersal in Iron Age Spain: a geometric morphometrics appraisal

    Biological Journal of the Linnean Society 102:483–497.

    https://doi.org/10.1111/j.1095-8312.2010.01603.x

    • Google Scholar
53. 1. Workman MS
    2. Leamy LJ
    3. Routman EJ
    4. Cheverud JM

    (2002)

    Analysis of quantitative trait locus effects on the size and shape of mandibular molars in mice

    Genetics 160:1573–1586.

    • PubMed
    • Google Scholar
54. 1. Wray NR
    2. Yang J
    3. Hayes BJ
    4. Price AL
    5. Goddard ME
    6. Visscher PM

    (2013) Pitfalls of predicting complex traits from SNPs

    Nature Reviews Genetics 14:507–515.

    https://doi.org/10.1038/nrg3457

    • PubMed
    • Google Scholar
55. 1. Yang H
    2. Ding Y
    3. Hutchins LN
    4. Szatkiewicz J
    5. Bell TA
    6. Paigen BJ
    7. Graber JH
    8. de Villena FP
    9. Churchill GA

    (2009) A customized and versatile high-density genotyping array for the mouse

    Nature Methods 6:663–666.

    https://doi.org/10.1038/nmeth.1359

    • PubMed
    • Google Scholar
56. 1. Yang J
    2. Lee SH
    3. Goddard ME
    4. Visscher PM

    (2011) GCTA: a tool for genome-wide complex trait analysis

    The American Journal of Human Genetics 88:76–82.

    https://doi.org/10.1016/j.ajhg.2010.11.011

    • PubMed
    • Google Scholar
57. 1. Zhou X
    2. Stephens M

    (2012) Genome-wide efficient mixed-model analysis for association studies

    Nature Genetics 44:821–824.

    https://doi.org/10.1038/ng.2310

    • PubMed
    • Google Scholar

Thank you for submitting your article "Genomic regions controlling shape variation in the first upper molar of the house mouse" for consideration by eLife. Your article has been favorably evaluated by Patricia Wittkopp (Senior Editor) and three reviewers, one of whom, Craig T Miller (Reviewer #1), is a member of our Board of Reviewing Editors. The following individual involved in review of your submission has agreed to reveal their identity: Alistair R. Evans (Reviewer #3).

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

Summary:

Pallares and colleagues study mouse molar shape variation in two subspecies of house mice. Using a GWAS and 3D surface morphometric approach in hybrid mice, the authors identify five genomic regions associated with molar shape. One genomic region contains the candidate gene Mitf, and the authors show in lab mice with different mutant alleles of Mitf a role for Mitf in regulating molar shape. Overall this study presents the first GWAS for molar shape in natural populations, identifies the first genomic regions found to underlie quantitative variation in molar shape in nature, and identifies a specific gene (Mitf) regulating quantitative aspects of molar shape in the lab that appears to also regulate natural variation in tooth shape.

Essential revisions:

1) One potential concern is many of the Mitf mutant comparisons are done between mutant mice and a panel of non-sibling B6 mice. It seems possible that differences in genetic background in these mouse lines contribute to the differences in tooth shape. One nice argument against this is the authors observe differences in tooth shape between mice heterozygous and homozygous for the mi-vga9 allele. It would strengthen the authors' conclusions if in Table 1 they compared the mi-vga9/mi-vga9 mice not just to wild-type mice, but to the mi-vga9/+ mice. In Figure 5, these genotypes look quite different, but it would be better to also formally test the differences. Along these lines, can the authors comment on whether the mi-vga9/mi-vga9 and mi-vga9/+ mice were full siblings? If so, that would further strengthen the argument that Mitf, not genetic background differences are responsible.

2) Although finding actual mutations underlying the Mitf-associated QTL is beyond the scope of this study, can the authors at least comment on whether coding changes are found between the domesticus and musculus mice used in their study, or whether Mitf coding changes are present in the reference genome assemblies available for domesticus and musculus derived strains or populations?

3) Subsection “Association mapping”, third paragraph: Can the authors further justify the decision to pick arbitrary 500 kb intervals for most QTL? It seems they have data that would speak to patterns of LD in this sample, including genome-wide averages and within these QTL, so not sure whether 500 kb is an overly conservative estimate or not.

4) Subsection “Association mapping”, second paragraph: I was surprised that no genomic regions were significantly associated with size variation, given how polygenic and strong the signal appears for tooth shape. Was mouse size corrected for in mapping centroid size?

5) "The other mutant alleles do not exhibit osteopetrosis in homozygotes." Can the authors clarify for this statement, whether they mean osteopetrosis has not been reported in homozygotes for these alleles, or whether they (or other groups) looked for osteopetrosis in homozygotes? If the latter, the authors should provide references or data to back up this claim. If the former, the sentence should be edited. Same comment for “The evidence presented here for effect of Mitf on molar shape comes from mutations in a mouse laboratory strain, and it is therefore not equivalent to comparing the effect of naturally occurring alleles”.

6) Materials and methods: Only the first upper molar is analyzed. Why not, but the rational for this choice is never explained through the manuscript.

7) Geometric morphometrics: You used 10 landmarks to anchor the template. They are very important features to know because they impose some constrains, but there is no way for the readers to get this information.

For the full template, 1588 semi-landmarks are used and for the "wear-free" 1532. This is much denser in the second case than in the first given that all tips of cusps are removed. I don't understand why you increase the sampling.

8) Subsection “Tooth shape, mouse age and wear”. Your idea is that age is related to wear, as once erupted tooth shape doesn't change except by the effect of wear. In my opinion you should explain it to readers.

9) Wear is expected to be on PC1. People have used Burnaby-like procedure to get rid of such artifacts (many examples with fish) based on PC1 or expected shape changes. You decide to use a very different alternative approach (quite a disappointing one) by cutting cusp tips. Why not, but why? Because age impacts a lot of PCs and thus you consider that the effect of wear is spread all over the shape space? If so, I think you should explain your rational. Interestingly, age explain the same amount of variance with the 'wear-free' template but is less significant.

10) Finally you analyzed something very close to what is captured with 2D outline. The amount of 3D information you have in your sampling depend on the cusp height kept in the template. You didn't say anything about that. How did you decide the height at which you cut the tips?

11) I don't understand why you run separate Procrustes superimpositions (subsection “Functional evaluation of the candidate gene Mitf”, second paragraph). It adds a very complex way of comparing shape changes. I can understand that you were afraid that weird mutant twists your Procrustean space for GWAS but in my opinion, you could have added the mutant as supplementary observations into the hybrid GPA (i.e. superimposing each mutant to the hybrid mean shape). It will have allowed the comparison of shape changes using more classical tools of geometric morphometrics (angles between shape vectors, vector correlation etc.). In your approach, I don't get how the two can be centered and aligned to ensure that trivial variation are removed.

12) One additional remark on this method of comparing shape changes is that in order to get a threshold you consider the values obtained between PCs. PCs are orthogonal but not independent (subsection “Functional evaluation of the candidate gene Mitf”, last paragraph). The math ensures that they are orthogonal but nothing ensure that they are independent given the underlying generative processes (for ex genetics).

13) GWAS: The family structure in your sample is very strong and you correctly use linear mixed model to handle it but, as in your previous papers, running multiple univariate LMM on PCs is not the most powerful approach, but I understand that running multivariate LMM is challenging, but the cost is evident here as you don't have a very powerful design with 183 mice.

"To facilitate association mapping" but it is at the price of some power lost because PCs are not aligned to the underlying genetics.

Nonetheless, you correctly want to assess the effect size of best SNPs simultaneously on the full shape space (on all tangent coordinates), but why using only the first 18 PCs? Why estimating these effect sizes only on 86% variance? Did you consider that 14% are just analytical errors?

14) About that (subsection “Association mapping”, last paragraph) you say that you used the coefficient of determination but with multivariate data I guess you used an analogue based on distances (a ratio of sums of sum-of-squares) but there are other analogue based on classical multivariate statistics (Pillai etc.). It may be better to precise that it is procrustes distances based R2.

15) It is unclear that you used leave-one chromosome out approach to handle the pop structure in your LMM (subsection “Association mapping”, second paragraph).

16) SNP heritability: subsection “SNP heritability”, first paragraph: In my opinion, the way you present your computation of the "total heritability" is right in calculus but is little weird in term of genetics. In my opinion, you assess the sum of the additive genetic variances and do the ratio with the total variance to get a quite poor proxy of h2, and indeed means not much in term of heritability (see discussions of Monteiro and Klingenberg between 2002 and 2010).

17) Functional evaluation: subsection “Samples used to functionally evaluate Mitf”: I don't agree with this statement about sexual dimorphism. There are plenty of phenotypes with SD in mammals and more specifically in the house mouse (see for example Kart et al. 2016 Nat Comms). The fact that SD is low on bone shape doesn't insure that it is the case in tooth.

18) Results, first paragraph: The most parsimonious hypothesis is simply it is within species variation and that the inter-subspecific variance is smaller than the within, species-specific loci have small effects on this character and are not distinguishable from within-species loci. I don't think we need to ask for transgressive variation or hybrid instability.

19) Discussion: subsection “Genetic architecture”, first paragraph: Actually this conclusion is based on the huge amount of missing heritability (difference between the SNP heritability and the actual loci you catch-up). However, as your GWAS doesn't properly handle the multivariate nature of the shape space, you don't know anything about that because for instance a locus, orthogonal to all PCs, and explaining 1% of variance on each, will finally explained a lot of variance but will never be captured with your approach.

20) You have a very small sample size to run Hotelling T². Will you do a t-test with N = 5 samples? Same conditions apply to Hotelling. Moreover you have twice more additional parameters plus correlation between the two variables to estimate. Maybe doing something non-parametric based on distances will be more reliable.

21) Some p-values for statistical tests were not reported in the manuscript.

Essential revisions:

1) One potential concern is many of the Mitf mutant comparisons are done between mutant mice and a panel of non-sibling B6 mice. It seems possible that differences in genetic background in these mouse lines contribute to the differences in tooth shape. One nice argument against this is the authors observe differences in tooth shape between mice heterozygous and homozygous for the mi-vga9 allele. It would strengthen the authors' conclusions if in Table 1 they compared the mi-vga9/mi-vga9 mice not just to wild-type mice, but to the mi-vga9/+ mice. In Figure 5, these genotypes look quite different, but it would be better to also formally test the differences. Along these lines, can the authors comment on whether the mi-vga9/mi-vga9 and mi-vga9/+ mice were full siblings? If so, that would further strengthen the argument that Mitf, not genetic background differences are responsible.

All the Mitf mutations are on a B6 background (this is stated in the Materials and methods section “Samples used to functionally evaluate Mitf”), and the B6 line used to generate the Mitf mutations is the same one used as control; this line is maintained in the same facilities and under the same conditions than the Mitf mutant lines derived from it.

The mi-vga9/mi-vga9 and mi-vga9/+ were indeed full siblings. This information was added in the first paragraph of the subsection “Samples used to functionally evaluate Mitf”. We have also added the comparison between mi-vga9/mi-vga9 and mi-vga9/+ in Table 1, demonstrating that differences between homozygous and heterozygous siblings was significant (p = 0.0076).

2) Although finding actual mutations underlying the Mitf-associated QTL is beyond the scope of this study, can the authors at least comment on whether coding changes are found between the domesticus and musculus mice used in their study, or whether Mitf coding changes are present in the reference genome assemblies available for domesticus and musculus derived strains or populations?

We compared Mitf sequences in samples from 6 wild populations of mice, including M. m. musculus and M. m. domesticus, M. m. castaneus and related M. spretus (Harr et al. Scientific Data 3, Article number: 160075 (2016) doi:10.1038/sdata.2016.75), and eight hybrid mice from the same population used in this study (Turner Tautz & Harr, unpublished data).

We found seven predicted missense variants in wild populations of mice. Five are found in Mus spretus and Mus castaneus. One variant is present in a population of M.m. musculus from Kazakhstan, and one in a population of M.m. domesticus in Germany. Another variant is present in mice from the hybrid zone used in this study.

We have added this information in the Discussion (last paragraph), and added a table of the predicted variants as Supplementary file 1B.

The SNP data can be accessed in the UCSC genome browser (genome.ucsc.edu) -> My Data -> Public Sessions -> wildmouse

3) Subsection “Association mapping”, third paragraph: Can the authors further justify the decision to pick arbitrary 500 kb intervals for most QTL? It seems they have data that would speak to patterns of LD in this sample, including genome-wide averages and within these QTL, so not sure whether 500 kb is an overly conservative estimate or not.

In a previous study where we used the same SNP data (Pallares et al. 2014) we showed that the median size of the associated regions with LD > 0.8 was 150kb, and the regions with LD > 0.2 (weakly linked) was ~ 1.8 Mb. In the present study, some significant SNPs fell in regions with poor SNP density resulting in zero neighboring SNPs with LD> 0.8. Therefore, we decided to use the intermediate value 500 Kb here, similar to what we observed in the previous study, but not as stringent. We added a clarification of this choice in the third paragraph of the subsection “Association mapping”.

4) Subsection “Association mapping”, second paragraph: I was surprised that no genomic regions were significantly associated with size variation, given how polygenic and strong the signal appears for tooth shape. Was mouse size corrected for in mapping centroid size?

No, we didn’t account for the effect of body size on tooth centroid size. The first upper molar is mineralized around birth in mice, and although it might be correlated with body size at birth (we don’t have this data), it is not correlated with adult body size at the population level (Renaud et al. 2017, Mamm Biol). Using body size as a covariate might introduce variation that is actually not relevant for molar size. Besides, the lack of associations with size is not rare, usually such loci are of very small effect, and the studies that have been able to identify them are usually performed with F2 crosses of mouse inbred lines that were previously selected for body size differences.

5) "The other mutant alleles do not exhibit osteopetrosis in homozygotes." Can the authors clarify for this statement, whether they mean osteopetrosis has not been reported in homozygotes for these alleles, or whether they (or other groups) looked for osteopetrosis in homozygotes? If the latter, the authors should provide references or data to back up this claim. If the former, the sentence should be edited. Same comment for “The evidence presented here for effect of Mitf on molar shape comes from mutations in a mouse laboratory strain, and it is therefore not equivalent to comparing the effect of naturally occurring alleles”.

Osteopetrosis has been characterized for many of the Mitf mutations, the references for individual studies are found in the following review that we have now added as reference in the pertinent sections (subsection “Mitf gene affects molar shape in mice”, second, fourth and sixth paragraphs and subsection “Mitf and molar shape variation”, second paragraph): Steingrimsson et al., 2004, annual review genetics. And, we have also added, in the same sections, the following reference where the alleles mi, ew, Wh, and vga9 used in this study were directly tested for osteopetrosis: Steingrimsson et al. 2002, PNAS.

6) Materials and methods: Only the first upper molar is analyzed. Why not, but the rational for this choice is never explained through the manuscript.

We decided to focus on the first molar because it is the first one to develop in the molar row and therefore, it is relatively independent from the development of the second and third molars given that they develop following the Inhibitory-Cascade model (Kavanagh et al. Nature 2007). The second and third molar, in contrast, are not only a simplified version (in terms of morphological complexity) of the first molar, but are also directly affected/constrained by the development of the first one. Along these lines, the effect of genetic variation relative to environmental variation (in this case the development of other molars) on molar shape is enhanced on the first molar.

Regarding the lower molars, these are less morphologically complex compared to the upper molars. And therefore, exploring the first upper molar maximizes the amount of phenotypic variation available for mapping.

Given the results found here for the first upper molar we think it will be very interesting to explore the genetic basis of molar shape in the second and third molars independently of each other, but also in terms of the correlations that are created between them given the Inhibition-Cascade model. This however, will need to be explored in a different manuscript.

We added a line in the Introduction specifying that we focused on the first upper molar (fourth paragraph) and expanded on the reason for this in the second paragraph of the Materials and methods subsection “3D surface morphometrics”

7) Geometric morphometrics: You used 10 landmarks to anchor the template. They are very important features to know because they impose some constrains, but there is no way for the readers to get this information.

For the full template, 1588 semi-landmarks are used and for the "wear-free" 1532. This is much denser in the second case than in the first given that all tips of cusps are removed. I don't understand why you increase the sampling.

The 10 landmarks were only used to pre-position the template to the original template before this was deformed to match the tooth surface. These landmarks were not used in any analysis and therefore they should not affect any of the morphometric results and interpretations. We have, however, added the landmarks to the surfaces on Figure 2, so it is clear for the reader where are they located.

The semi-landmarks are automatically sampled following a Poisson distribution with the aim of obtaining equally-spaced semi-landmarks. The number requested was 1500 for both templates with the aim of obtaining a dense sampling and therefore capture variation at a very small scale. To fulfill the equally-spaced requirement, the actual number selected by the algorithm is approximately the requested number, therefore the specific numbers used in each template are not identical. Since the number of semi-landmarks is very high, going from, e.g. 1500 in the full template to e.g. 1000 in the wear-free template won’t affect the observed results, and therefore we opted for using the same number for both.

We have now added in the Materials and methods, a statement regarding the number of requested landmarks requested and the number finally sampled (subsection “3D surface morphometrics”, third and fifth paragraphs).

8) Subsection “Tooth shape, mouse age and wear”. Your idea is that age is related to wear, as once erupted tooth shape doesn't change except by the effect of wear. In my opinion you should explain it to readers.

We have expanded the Results section “Tooth shape, mouse age and wear”to include the explanation for this. In short, yes, tooth shape doesn’t change once the teeth have erupted. Only wear affects tooth morphology after eruption, and this is why wear and age are correlated. However, other non-age-related factors like diet and behavior also affect the degree of wear, and this is one of the reasons why we used a wear-free template approach, to be able to address not only age-related wear patterns, but also other unknown factors.

9) Wear is expected to be on PC1. People have used Burnaby-like procedure to get rid of such artifacts (many examples with fish) based on PC1 or expected shape changes. You decide to use a very different alternative approach (quite a disappointing one) by cutting cusp tips. Why not, but why? Because age impacts a lot of PCs and thus you consider that the effect of wear is spread all over the shape space? If so, I think you should explain your rational. Interestingly, age explain the same amount of variance with the 'wear-free' template but is less significant.

Wear is a very difficult aspect to address especially because, as correctly suggested by the reviewers, it impacts several PCs. Age, although correlated with wear is an incomplete proxy for wear since there are other factors affecting wear patterns as described in the response to the previous comment and explored in Renaud and Ledevin 2017. Before choosing the wear-free method, we tried several approaches such as multivariate regressions, or using the dentine-enamel interface as the surface of interest. At the end, removing the cusp tips appeared to be the most convincing protocol. We also considered the fact that it should be possible to apply this phenotyping method to wild animals for which age is unknown (e.g. Ledevin et al., 2016).

10) Finally you analyzed something very close to what is captured with 2D outline. The amount of 3D information you have in your sampling depend on the cusp height kept in the template. You didn't say anything about that. How did you decide the height at which you cut the tips?

It is true that traditional 2D and our 3D phenotyping seem similar as they both focus on the crown, however, our 3D approach detects shape details that a 2D outline will not be able to detect. It captures information from the crown outline but also from the central parts of the tooth such as the inter-cusp space and the morphology of the cusp basal area. The slope at the front of the cusps, described by our 3D method, is not accessible with a 2D outline.

The height at which we cut the tip was decided empirically by comparing teeth of different wear stages. The cut section was designed so that the most worn teeth of our dataset can still be described by our template. We have now added this in the Materials and methods section (subsection “3D surface morphometrics”, fifth paragraph).

11) I don't understand why you run separate Procrustes superimpositions (subsection “Functional evaluation of the candidate gene Mitf”, second paragraph). It adds a very complex way of comparing shape changes. I can understand that you were afraid that weird mutant twists your Procrustean space for GWAS but in my opinion, you could have added the mutant as supplementary observations into the hybrid GPA (i.e. superimposing each mutant to the hybrid mean shape). It will have allowed the comparison of shape changes using more classical tools of geometric morphometrics (angles between shape vectors, vector correlation etc.). In your approach, I don't get how the two can be centered and aligned to ensure that trivial variation are removed.

Adding mutants as supplementary specimens is an option that we considered, however the probability for mutants would have been outside of the range of variation of the wild-derived mice. The risk of incorrect projection of the mutants into the shape space was therefore not negligible, and we preferred to run two separate analyses. Now, since there is not a Procrustes superimposition, there is not centering and alignment, and therefore our method doesn’t compare effects in the shape space, but directly on the tooth surface. The question we tried to answer in this way is whether two effects (e.g. mutant alleles, SNPs from the mapping) modify the tooth in the same way.

12) One additional remark on this method of comparing shape changes is that in order to get a threshold you consider the values obtained between PCs. PCs are orthogonal but not independent (subsection “Functional evaluation of the candidate gene Mitf”, last paragraph). The math ensures that they are orthogonal but nothing ensure that they are independent given the underlying generative processes (for ex genetics).

We have now re-written this part to make clear that the phenotypic changes associated with PCs are not necessarily independent given the underlying genetics. However, since it is not feasible to get totally independent shape changes to be used as a null in Figure 5, we still use the correlation between PCs as a reference for the correlation between orthogonal shape changes. We have however, been very clear regarding that a genetic correlation might exist between PC, and therefore we are not using such correlations as a “significance threshold” as was stated previously in the text, but just as a point of comparison. (subsection “Functional evaluation of the candidate gene Mitf”, last paragraph and legend Figure 5).

13) GWAS: The family structure in your sample is very strong and you correctly use linear mixed model to handle it but, as in your previous papers, running multiple univariate LMM on PCs is not the most powerful approach, but I understand that running multivariate LMM is challenging, but the cost is evident here as you don't have a very powerful design with 183 mice.

"To facilitate association mapping" but it is at the price of some power lost because PCs are not aligned to the underlying genetics.

Nonetheless, you correctly want to assess the effect size of best SNPs simultaneously on the full shape space (on all tangent coordinates), but why using only the first 18 PCs? Why estimating these effect sizes only on 86% variance? Did you consider that 14% are just analytical errors?

We agree with this assessment. The mapping of univariate PCs is limited, and it is not the optimal way of mapping shape data. Whenever possible, shape data should be mapped in a multivariate manner. However, this was the way to circumvent the strong family structure of the data and therefore to be able to implement the LMM. We have now mentioned the limitations of this approach (subsection “Association mapping”, first paragraph) and removed the line “to facilitate association mapping”.

The choice of the first 18 PCs was totally arbitrary, as it usually is when it comes to decide on the “relevant” PCs. Here we just chose as a cutoff, the PCs that explained at least 1% of the variance.

14) About that (subsection “Association mapping”, last paragraph) you say that you used the coefficient of determination but with multivariate data I guess you used an analogue based on distances (a ratio of sums of sum-of-squares) but there are other analogue based on classical multivariate statistics (Pillai etc.). It may be better to precise that it is procrustes distances based R2.

Yes, we calculated r2 as the explained variance from sums of squares summed over all responses. We have clarified this in the third paragraph of the subsection “Association mapping”.

15) It is unclear that you used leave-one chromosome out approach to handle the pop structure in your LMM (subsection “Association mapping”, second paragraph).

We didn’t use the leave-one-chromosome-out approach. We used a unique kinship matrix including all chromosomes to evaluate associations in each chromosome. We however acknowledge that excluding the focal chromosome might have increased our power a bit.

16) SNP heritability: subsection “SNP heritability”, first paragraph: In my opinion, the way you present your computation of the "total heritability" is right in calculus but is little weird in term of genetics. In my opinion, you assess the sum of the additive genetic variances and do the ratio with the total variance to get a quite poor proxy of h2, and indeed means not much in term of heritability (see discussions of Monteiro and Klingenberg between 2002 and 2010).

The specific way in which heritability was estimated in this study is again, as a comment above adequately pointed out, a limitation associated with performing univariate mapping with a multivariate trait, and we acknowledge that. However, we don’t completely agree with the idea that estimating heritability for each shape dimension is the right approach for the question we want to answer in this study (referring to the Monteiro and Klingenberg discussion). We think that depending on the objective of the study (as we think it was the conclusion of Monteiro and Klingenberg discussion), a univariate estimate of variance is adequate. If the objective is, for example, to address response to selection, a scalar heritability value is of little use. But, if, as in this study, the intention is to get an impression on how much additive genetic variance underlies a trait, a scalar value is the only way of making it comparable with other studies or traits.

We have now clarified the intention behind calculating a scalar value for heritability (subsection “SNP heritability”, first paragraph).

17) Functional evaluation: subsection “Samples used to functionally evaluate Mitf”: I don't agree with this statement about sexual dimorphism. There are plenty of phenotypes with SD in mammals and more specifically in the house mouse (see for example Kart et al. 2016 Nat Comms). The fact that SD is low on bone shape doesn't insure that it is the case in tooth.

We totally agree with this comment, and with the results shown in the paper suggested by the reviewers. By no means we wanted to imply that sexual dimorphism, in general, is not important in mice. Thanks to the reviewer comment, we noticed that we had stated “sex has very small effect in bone shape”, however we meant tooth shape, and not bone shape. There’s evidence (we have cited such references, e.g. Valenzuela-Lamas et al. 2011, Renaud et al. 2017) that sexual dimorphism is very small, and most of the time is absent in molar tooth shape. Therefore, although our sample size is small and doesn’t allow us to test for sexual dimorphism, we are still confident that such factor doesn’t affect our results. We have now corrected the text (subsection “Samples used to functionally evaluate Mitf”, last paragraph).

18) Results, first paragraph: The most parsimonious hypothesis is simply it is within species variation and that the inter-subspecific variance is smaller than the within, species-specific loci have small effects on this character and are not distinguishable from within-species loci. I don't think we need to ask for transgressive variation or hybrid instability.

We agree with this interpretation. We have now added it to the text (Results, first paragraph), including reference where such pattern was described for bone shape. We have, however, kept transgressive phenotypes and hybrid instability as other sources of within-pop variation given the hybrid nature of the sample used in this study.

19) Discussion: subsection “Genetic architecture”, first paragraph: Actually this conclusion is based on the huge amount of missing heritability (difference between the SNP heritability and the actual loci you catch-up). However, as your GWAS doesn't properly handle the multivariate nature of the shape space, you don't know anything about that because for instance a locus, orthogonal to all PCs, and explaining 1% of variance on each, will finally explained a lot of variance but will never be captured with your approach.

Our conclusion has two lines of evidence. First, as the reviewers mention, the large amount of missing heritability. And second, the results derived from the chromosomal partitioning of the variance shown in Figure 3.

We agree with the fact that our univariate mapping approach misses genetic associations that are spread across many PCs, and this will affect the missing heritability estimates. However, the percentage of variance explained by a locus increases, but not in a dramatic way, when it is calculated based on the PC it is associated with, vs. on the combined effect of all PCs. We therefore, don’t expect single loci to explain a lot of variance in this phenotype, even if multivariate approach were used.

Yet, we offer another line of evidence for the polygenicity of tooth shape that is totally independent of the associations found in GWAS: the results of the chromosomal partitioning of the variance. This test has been used, mostly in human studies, to determine “how polygenic a polygenic trait is”. In this test, no individual associations between SNPs and phenotype are calculated, but the joint effect of all SNPs found in the different chromosomes. The rationale behind this is: if the effect of individual SNPs is small and many SNPs underlie phenotypic variation, there will be a linear correlation between chromosomal length and phenotypic effect. This is exactly the pattern that we see for tooth shape, and therefore, although we acknowledge and share the reviewers concerns regarding the first line of evidence, we are confident on the interpretation of our results.

We have added lines in the Results section to make clear the importance of the chromosomal-partitioning-of-the-variance analysis.

20) You have a very small sample size to run Hotelling T². Will you do a t-test with N = 5 samples? Same conditions apply to Hotelling. Moreover you have twice more additional parameters plus correlation between the two variables to estimate. Maybe doing something non-parametric based on distances will be more reliable.

It is always tricky to test for differences when the sample size is small. However, given the circumstances (it was not possible to obtain more mutant mice), a t-test (here a T²) is one of the most performant tests given a small sample size (e.g. de Winter, PARE, 2013). More important than the significance of this test, is the distribution of the different mutant shapes that show a clear effect of each of the mutant alleles on molar tooth shape.

21) Some p-values for statistical tests were not reported in the manuscript.

We have now added the missing p-values in the last paragraph of the subsection “Tooth shape, mouse age and wear”, and subsection “Heritability”.

Author details

1. Luisa F Pallares

   Department of Evolutionary Genetics, Max-Planck Institute for Evolutionary Biology, Plön, Germany

   Present address

   Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, United States

   Contribution

   Conceptualization, Formal analysis, Investigation, Writing—original draft, Writing—review and editing

   Contributed equally with

   Ronan Ledevin

   For correspondence

   pallares@princeton.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-6547-1901

2. Ronan Ledevin

   Laboratoire de Biométrie et Biologie Evolutive, UMR5558, CNRS, University Lyon 1, Campus de la Doua, Villeurbanne, France

   Present address

   Université de Bordeaux, PACEA, UMR5199, 33615 Pessac, France

   Contribution

   Conceptualization, Formal analysis, Investigation, Writing—review and editing

   Contributed equally with

   Luisa F Pallares

   Competing interests

   No competing interests declared

3. Sophie Pantalacci

   ENS de Lyon, Univ Claude Bernard, CNRS UMR 5239, INSERM U1210, Laboratoire de Biologie et Modélisation de la Cellule, 15 parvis Descartes, F-69007, UnivLyon, Lyon, France

   Contribution

   Conceptualization, Writing—review and editing

   Competing interests

   No competing interests declared

4. Leslie M Turner

   1. Department of Evolutionary Genetics, Max-Planck Institute for Evolutionary Biology, Plön, Germany
   2. Department of Biology and Biochemistry, Milner Centre for Evolution, University of Bath, Bath, Unites States

   Contribution

   Conceptualization, Resources, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-5105-3546

5. Eirikur Steingrimsson

   Department of Biochemistry and Molecular Biology, BioMedical Center, Faculty of Medicine, University of Iceland, Reykjavik, Iceland

   Contribution

   Conceptualization, Resources, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5826-7486

6. Sabrina Renaud

   Laboratoire de Biométrie et Biologie Evolutive, UMR5558, CNRS, University Lyon 1, Campus de la Doua, Villeurbanne, France

   Contribution

   Conceptualization, Supervision, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-8730-3113

• 1,157

  Page views

• 148

  Downloads

• 5

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, Scopus, PubMed Central.