Biases of Principal Component Analysis (PCA) in Physical Anthropology Studies Require a Reevaluation of Evolutionary Insights

Reviewer #1 (Public review):

Mohseni and Elhaik have critically examined the widespread use of principal component analysis (PCA) in phylogenetic inferences within the discipline of physical anthropology. The authors present compelling evidence that PCA underperforms compared to machine learning (ML) classifiers. This excellent work not only challenges the reliability of PCA-based taxonomic inferences, but also adds to a growing body of literature questioning the application of PCA in physical anthropology, thereby initiating a fruitful discussion in our field. Moreover, it underscores the crucial need of external validation methods in such studies.

The authors have addressed nearly all of my comments, and my questions have been fully answered. The revised manuscript represents a significant improvement.

The new title more effectively conveys the central message emerging from this research; The revised introduction more precisely addresses the methodological challenges currently facing the discipline.
I am equally amazed by the profound susceptibility of the PCA results, as demonstrated by the alterations introduced by the authors, and by the contrasting robustness of the ML classifiers. I trust that this contrast will spark a fruitful discussion about the application of both methods in our field. It should also inspire further research conducted by physical anthropologists to study the role of ML in this discipline.
Lastly, and importantly, I believe the authors should be commended for addressing the broader implications of their work, particularly in relation to public perceptions of science (pp. 20-21).

Reviewer #3 (Public review):

Mohseni and Elhaik challenge the widespread use of PCA as an analytical and interpretive tool in the study of geometric morphometrics. The standard approach in geometric morphometrics analysis involves Generalised Procrustes Analysis (GPA) followed by Principal Component Analysis (PCA). Recent research challenges PCA outcomes' accuracy, robustness, and reproducibility in morphometrics analysis. In this paper, the authors demonstrate that PCA is unreliable for such studies. Additionally, they test and compare several Machine-Learning methods and present MORPHIX, a Python package of their making that incorporates the tools necessary to perform morphometrics analysis using ML methods.

Mohseni and Elhaik conducted a set of thorough investigations to test PCA's accuracy, robustness, and reproducibility following renewed recent criticism and publications where this method was abused. Using a set of 2 and 3D morphometric benchmark data, the authors performed a traditional analysis using GPA and PCA, followed by a reanalysis of the data using alternative classifiers and rigorous testing of the different outcomes.

In the current paper, the authors evaluated eight ML methods and compared their classification accuracy to traditional PCA. Additionally, common occurrences in the attempted morphological classification of specimens, such as non-representative partial sampling, missing specimens, and missing landmarks, were simulated, and the performance of PCA vs ML methods was evaluated.

Comments on revisions:

I have gone over the revised manuscript and the detailed responses to the previous round of review. While there are places where I personally would have used slightly toned-down phrasing, the authors' get to set the tone of their manuscript, and I will not argue with that any further.

In general, the restructuring, addition of new paragraphs, minor revisions and new title make for a much better manuscript, which as stated in the previous review, will be a valuable resource for workers in the field.

Author response:

The following is the authors’ response to the original reviews.

Reviewer #1:

Comment 1. Mohseni and Elhaik's article offers a critical evaluation of Geometric Morphometrics (GM), a common tool in physical anthropology for studying morphological differences and making phylogenetic inferences. I read their article with great interest, although I am not a geneticist or an expert on PCA theory since the problem of morphology-based classification is at the core of paleoanthropology.

The authors developed a Python package for processing superimposed landmark data with classifier and outlier detection methods, to evaluate the adequacy of the standard approach to shape analysis via modern GM. They call into question the accuracy, robustness, and reproducibility of GM, and demonstrate how PCA introduces statistical artefacts specific to the data, thus challenging its scientific rigor. The authors demonstrate the superiority of machine learning methods in classification and outlier detection tasks. The paper is well-written and provides strong evidence in support of the authors' argument. Thus, in my opinion, it constitutes a major contribution to the field of physical anthropology, as it provides a critical and necessary evaluation of what has become a basic tool for studying morphology, and of the assumptions allowing its application for phylogenetic inferences. Again, I am not an expert in these statistical methods, nor a geneticist, but the authors' contribution is of substantial relevance to our field (physical anthropology). The examples of NR fossils and HLD 6 are cases in point, in line with other notable examples of critical assessment of phylogenetic inferences made on the basis of PCA results of GM analysis. For example, see Lordkipanidze et al.'s (2014) GM analyses of the Dmanisi fossils, suggesting that the five crania represent a single regional variant of Homo erectus; and see Schwartz et al.'s (2014) comment on their findings, claiming that the dental, mandibular, and cranial morphology of these fossils suggest taxic diversity. Schwartz et al. (2014) ask, "Why did the GMA of 78 landmarks not capture the visually obvious differences between the Dmanisi crania and specimens commonly subsumed H. erectus? ... one wonders how phylogenetically reliable a method can be that does not reflect even easily visible gross morphological differences" (p. 360).

As an alternative to the PCA step in GM, the authors tested eight leading supervised learning classifiers and outlier detection methods on three-dimensional datasets. The authors demonstrated inconsistency of PCA clustering with the taxonomy of the species investigated for the reconstruction of their phylogeny, by analyzing a database comprising landmarks of 6 known species that belong to the Old World monkeys tribe Papionini, using PCA for classification. The authors also demonstrated that high explained variance should not be used as an estimate of high accuracy (reliability). Then, the authors altered the dataset in several ways to simulate the characteristic nature of paleontological data.

The authors excluded taxa from the database to study how PCA and alternative classifiers are affected by partial sampling, and the results presented in Figures 4 and 5, among others, are quite remarkable in showing the deviations from the benchmark data. These results expose the perils of applying PCA and GM for interpreting morphological data. Furthermore, they provide evidence showing that the alternative classifiers are superior to PCA, and that they are less susceptible to experimenter intervention. Similar results, i.e., inconsistencies in the PC plots, were obtained in examinations of the effect of removing specimens from the dataset and in the interesting test of removing landmarks to simulate partial morphological data, as is often the case with fossils. To test the combined effect of these data alterations, the authors combined removal of taxa, specific samples, and landmarks from the dataset. In this case, as well, the PCA results indicate deviation from the benchmark data. However, the ML classifiers could not remedy the situation. The authors discuss how these inconsistencies may lead to different interpretations of the data, and in turn, different phylogenetic conclusions. Lastly, the authors simulated the situation of a specimen of unknown taxonomy using outlier detection methods, demonstrating LOF's ability to identify a novelty in the morphospace.

References

Bookstein FL. 1991. Morphometric tools for landmark data: geometry and biology [Orange book]. Cambridge New York: Cambridge University Press.
Cooke SB, and Terhune CE. 2015. Form, function, and geometric morphometrics. The Anatomical Records 298:5-28.
Lordkipanidze D, et al. 2013. A complete skull from Dmanisi, Georgia, and the evolutionary biology of early Homo. Science 342: 326-331.
Schwartz JH, Tattersall I, and Chi Z. 2014. Comment on "A complete skull from Dmanisi, Georgia, and the evolutionary biology of Early Homo". Science 344(6182): 360-a.

The reviewer considered our work to be a “contribution is of substantial relevance to our field (physical anthropology)” We are grateful for this evaluation and for the thorough review and insightful comments on our manuscript, which helped us improve its quality further. Your remarks regarding the superiority of machine learning methods over traditional GM approaches, as well as the challenges and implications highlighted in our findings, resonate deeply with the core objectives of our research. The references to previous studies and their relevance to our work underscore the broader implications of our findings for the interpretation of morphological data in evolutionary studies. We are thankful for your remarks regarding the debate surrounding the Dmanisi fossils. We covered it in our introduction (lines 161-174):

Finally, PCA also played a part in the much-disputed case of the Dmanisi hominins (39, 40). These early Pleistocene hominins, whose fossils were recovered at Dmanisi (Georgia), have been a subject of intense study and debate within physical anthropology. Despite their small brain size and primitive skeletal architecture, the Dmanisi fossils represent Eurasia’s earliest well-dated hominin fossils, offering insights into early hominin migrations out of Africa. The taxonomic status of the Dmanisi hominins has been initially classified as Homo erectus or potentially represented a new species, Homo georgicus or else (40, 41). Lordkipanidze et al.’s (42) geometric morphometrics analyses suggested that the variation observed among the Dmanisi skulls may represent a single regional variant of Homo erectus. However, Schwartz et al. (2014) (43) raised concerns about the phylogenetic inferences based on PCA results of the geometric morphometrics analysis, noting the failure of the method to capture visually obvious differences between the Dmanisi crania and specimens commonly subsumed under Homo erectus."

Comment 2. I suggest moving all the interpretations from the Results section to the Discussion section. This will enhance the flow of the results and make it easier to follow.

We tried that, but it made the manuscript less readable. Because our manuscript makes two strong statements, one about the unsuitability of PCA to the field and one about the many other problems in the field, as demonstrated through several test cases, it is better to keep them separate in the Results and Discussions, respectively.

Comment 3. I recommend conducting an English language edit on the text to address minor inconsistencies.

We thoroughly edited the text to enhance the language style and consistency. We thank the reviewer for the suggestion.

Comment 4. Line 21, what do you mean by "ontogenists"?

Individuals who are versed in or study ontogeny.

Comment 5. When referring to the remains from Nesher Ramla (Israel), I recommend using "NR fossils". Thus, in line 34, I suggest replacing "Homo Nesher Ramla" by "Nesher Ramla fossils (NR fossils)", also in line 122.

We replaced "Homo Nesher Ramla" with "Nesher Ramla fossils (NR fossils)" in all of the instances throughout the manuscript. We thank the reviewer for the suggestion.

Comment 6. Line 34, I suggest replacing "human" by "hominin".

(Line 35) We replaced "human" with "hominin".

“…, such as the case of Homo Nesher Ramla, an archaic hominin with a questionable taxonomy.”

We thank the reviewer for the suggestion.

Comment 7. Line 67-68, I suggest clarifying the classification of landmarks using the definition of landmark types (Bookstein, 1991; also see summary by Cooke and Terhune (2015) - Table 1).

We revised our summary of the classification of landmarks: (Lines 83-94). Our MS now reads:

“Determining sufficient measurements and data points for a valid morphometric analysis is older than modern geometric morphometrics (19). In geometric morphometrics, landmarks are discrete points on biological structures used to capture shape variation. Bookstein (20) categorised landmarks into three types: Type one, representing the juxtaposition of tissues such as the intersection of two sutures; Type two, denoting maxima of curvature like the deepest point in a depression or the most projecting point on a process; and Type three, which includes extremal points defined by information from other locations on the object, such as the endpoint or centroid of a curve or feature. Originally, Type three landmarks encompassed semi-landmarks, but Weber and Bookstein (21) refined this classification, identifying Type three landmarks as those characterised by information from multiple curves and symmetry, including the intersection of two curves or the intersection of a curve and a suture, and further subdividing them into three subtypes (3a, 3b, 3c) (15). While landmarks provide crucial information about the structure’s overall shape, semi-landmarks capture fine-scale shape variation (e.g., curves or surfaces) that landmarks alone cannot adequately represent. Semi-landmarks are heavily relied upon as the source of shape information to break the continuity of regions in the specimen without clearly identifiable landmarks (22). Semi-landmarks are typically aligned based on their relative positions to landmarks, allowing for the comprehensive analysis of shape changes and deformations within complex structures (2). Unsurprisingly, the use of semi-landmarks is controversial. For instance, Bardua et al. (23) claim that high-density sliding semi-landmark approaches offer advantages compared to landmark-only studies, while Cardini (24) advises caution about potential biases and subsequent inaccuracies in high-density morphometric analyses.”

We thank the reviewer for the suggestion.

Comment 8. Line 84, "beneficial over" - I suggest revising.

(Line 102) We revised the sentence and used “offer advantages” instead.

“… claim that high-density sliding semi-landmark approaches offer advantages compared to landmark-only studies.”

We thank the reviewer for the suggestion.

Comment 9. Line 97, do you mean "therefore"?

(Line 115) Yes, we replaced "thereby" with "therefore".

Comment 10. Line 116, I suggest rephrasing as follows: "newly discovered hominin fossils with respect to...".

(Lines 135, 136) We rephrased it as suggested:

“is the classification of newly discovered hominin fossils within the human phylogenetic tree”

We thank the reviewer for the suggestion.

Comment 11. Line 119, please clarify or explain what you mean by subjective determination of clustering in PCA plots.

We rephrased (Lines 137, 138) to read:

"However, which specimens should be included in clusters and which ones should be considered outliers is determined subjectively…"

We thank the reviewer for the suggestion.

Comment 12. Lines 146-148: consider revising to clarify the sentence; "than" in line 147 should be "that".

We modified the sentence, we replaced "than" with "that". (Lines 196, 197)

" … that even the criticism from its pioneers was dismissed"

We thank the reviewer for the suggestion.

Comment 13. Line 213: I recommend adding the phylogenetic tree of the Papionini tribe. This would be particularly relevant for the interpretation of the results, e.g., in lines 324-328.

The reviewer suggested adding a phylogenetic tree of the Papionini tribe to increase the interpretability of our results. We added two trees (Figure 3) based on the molecular phylogeny of extant papionins and the most parsimonious tree generated from the initial Collard and Wood (1).

We thank the reviewer for the suggestion.

Comment 14. Lines 244-248: I recommend that the parallels drawn between the results presented in this section and other cases of PCA analysis interpretation (e.g., the NR fossils) are transferred to the Discussion section.

This would allow a more fluent read of the results.

Thank you, we considered that but found that it does not improve the readability of the discussion, because this is a very technical issue that would be best understood alongside the specific use case that tests it.

Comment 15. Line 301: The word "are" should be placed before the word "all".

(Line 319) We modified accordingly and placed "are" before "all":

“Rarely are all related taxa represented;”

We thank the reviewer for the suggestion.

Comment 16. Line 426: I suggest "omissions" in place of "missingness".

(Line 435) We replaced "missingness" with "omissions".

We thank the reviewer for the suggestion.

Comment 17. Line 440 is part of the caption for Figure 6. Please add a description of what the red arrow indicates in every figure in which it appears.

Yes, we added a sentence to the caption of figures 7 and 8:

“The red arrow in subfigures A, B, and C marks a Lophocebus albigena (pink) sample whose position in PC scatterplots is of interest.”

We thank the reviewer for the suggestion.

Comment 18. Line 454: I recommend "partial morphological information" instead of "some form information".

(Lines 446, 447) We made modifications and replaced "some form information" with " partial morphological information":

“Newfound samples often comprise incomplete osteological remains or fossils (18, 22) and only present partial morphological information.”

We thank the reviewer for the suggestion.

Comment 19. Line 547: I suggest "portion" instead of "fracture".

(Lines 470, 471) We replaced "fracture" with "portion":

“Thereby, while the complete skull would cluster with its own taxon…”

We thank the reviewer for the suggestion.

Comment 20. Lines 664-665 should read "anatomy and physical anthropology".

(Lines 600-602) We modified the text accordingly:

“There are various approaches in morphometrics, but among them, geometric morphometrics has left an indelible mark on biology, especially in anatomy and physical anthropology.”

We thank the reviewer for the suggestion.

Comment 21. Lines 684-699: This paragraph seems to belong in the introduction section.

(lines 175-190) We modified it and moved it to the introduction.

“Visual interpretations of the PC scatterplots are not the only role PCA plays in geometric morphometrics. Phylogenetic Principal Component Analysis (Phy-PCA) (44) and Phylogenetically Aligned Component Analysis (PACA) (45) are both used in geometric morphometrics to analyse shape variation while considering the supposed phylogenetic relationships among species. They differ in their approach to aligning landmark configurations and the role of PCA within them. Phy-PCA incorporates phylogenetic information by utilising a phylogenetic tree to model the evolutionary history of the species. This method aims to separate shape variation resulting from shared evolutionary history from other sources of variation. PCA plays a similar role in performing dimensionality reduction on the aligned landmark configurations in Phy-PCA (44). PACA takes a different approach to alignment. It uses a Procrustes superimposition method based on a phylogenetic distance matrix, aligning the landmark configurations according to the evolutionary relationships among species. PCA is then applied to the aligned configurations to extract the principal components of shape variation (45). Both analyses provide insights into the patterns and processes that shape biological form diversity while considering phylogenetic relationships, yet they are also subjected to the limitations and biases inherent in relying on PCA as part of the process.”

We thank the reviewer for the suggestion.

Comment 22. Line 717: I suggest "fossils" instead of "hominins".

(Lines 636, 637) We modified it accordingly and replaced "hominins" with "fossils":

“…which reflect the restraints faced in morphometric analysis of ancient samples (e.g., fossils).”

We thank the reviewer for the suggestion.

Comment 23. Line 728: the word "the" should be deleted; Skhul V should not be italicized, and so do the words "Mount Carmel"; "Neandertals"; "modern humans"; and "Late Paleolithic" in the following lines.

(Line 647-651) We made modifications accordingly:

“For example, Harvati (27), who analysed the Skhul 5 (84), a 40,000-year-old human skull from Mount Carmel (Israel), proposed diverging hypotheses based on favourable PC outcomes (based on PC8 separating it from Neanderthals and modern humans and associating it with the Late Palaeolithic specimen and based on PC12 associating it with modern humans).”

We thank the reviewer for the suggestion.

Comment 24. Line 734: the first comma should be deleted.

(Line 653) We deleted the first comma:

“(Figures 5-12) show that compared to the benchmark (Figure 4), …”

We thank the reviewer for the suggestion.

Reviewer #2:

Comment 1. I completely agree with the basic thrust of this study. Yes, of course, machine learning is FAR better than any variant of PCA for the paleosciences. I agree with the authors' critique early on that this point is not new per se - it is familiar to most of the founders of the field of GMM, including this reviewer. A crucial aspect is the dependence of ALL of GMM, PCA or otherwise, on the completely unexamined, unformalized praxis by which a landmark configuration is designed in the first place. I must admit that I am stunned by the authors' estimate of over 32K papers that have used PCA with GMM.

We thank the reviewer for accepting the premise of our study.

But beating a dead horse is not a good way of designing a motor vehicle. I think the manuscript needs to begin with a higher-level view of the pathology of its target disciplines, paleontology and paleoanthropology, along the lines that David demonstrated for numerical taxonomy some decades ago. That many thousands of bad methodologies require some sort of explanation all of their own in terms of (a) the fears of biologists about advanced mathematics, (b) the need for publications and tenure, (c) the desirability of covers of Nature and Science, and (d) the even greater glory of getting to name a new "species." This cumulative pathology of science results in paleoanthro turning into a branch of the humanities, where no single conclusion is treated as stable beyond the next dig, the next year or so of applied genomics, and the next chemical trace analysis. In short, the field is not cumulative.

Given the wide popularity of PCA and the attempts to prevent data replication to show its limitations, we do not believe that we are beating a dead horse, but a very live beast that threatens the integrity of the entire field. We accept the second part of the analogy about developing a motor vehicle.

We also accepted the reviewer’s suggestion and developed the suggested paragraph:

" A major contribution to the field was made by Sokal and Sneath’s Principles of Numerical Taxonomy (9) book, which challenged traditional taxonomic theory as inherently circular and introduced quantitative methods to address questions of classification (see also review by Sneath (10)). Hull (11) claimed that evolutionary reasoning practiced in taxonomy is not inherently circular but rather unwarranted. He argued that such criticism was based on misunderstandings of the logic of hypothesising, which he attributed to an unrealistic desire for a mistake-proof science. He contended that scientific hypotheses should begin with insufficient evidence and be refined iteratively as new evidence emerges. However, some taxonomists preferred a more rigid, hierarchical approach to avoid the appearance of error. As a result of these and other criticisms, traditional taxonomy declined in favour of cladistics and molecular systematics, which provided more accurate and evolutionarily informed classifications.

Today, palaeontology and palaeoanthropology grapple with methodological challenges that compromise the stability of their conclusions. These issues stem from various factors, including biologists’ apprehensions towards advanced mathematics, the pressure to publish for career advancement (12), the pursuit of high-profile journal covers, and the prestige associated with naming new species. As a result, these fields often resemble a branch of biology where the latest discoveries or new analytical techniques frequently overturn previous findings. This lack of cumulative knowledge necessitates a more rigorous approach to methodology and interpretation in morphometrics to ensure that conclusions are robust and enduring."

It is not obvious that the authors' suggestion of supervised machine learning will remedy this situation, since (a) that field itself is undergoing massive changes month by month with the advent of applications AI, and even more relevant (b) the best ML algorithms, those based on deep neural nets, are (literally) unpublishable - we cannot see how their decisions have actually been computed. Instead, to stabilize, the field will need to figure out how to base its inferences on some syntheses of actual empirical theories.

We appreciate the reviewer’s insightful comments and concerns regarding the use of supervised machine learning in our study. We acknowledge the rapid advancements in the field of machine learning and its significant impact on various domains, including geometric morphometrics. Although we are aware of the ongoing integration of machine learning techniques in geometric morphometrics, our objective was to thoroughly investigate some of the conventional and more frequently used models for comparative analysis.

Our intention was also to develop a Python module that enables users to easily apply these models to their landmark data. We recognise that most users typically apply machine learning methods to the principal component analysis (PCA) of their landmark data (2), unless PCA fails to explain enough variance (3), as we discussed in the context of Linear Discriminant Analysis (LDA). Our study demonstrates that these machine learning methods can be directly applied after generalised Procrustes analysis (GPA), without necessitating PCA as an intermediary step. This highlights another significant point of our research: the often automatic and potentially unnecessary use of PCA in geometric morphometrics.

Furthermore, we acknowledge that the availability of more extensive data might have allowed us to explore more complex methods, such as neural networks. However, neural networks require a substantial amount of data due to their numerous learning parameters, which we did not possess in this study. It is also evident that not every algorithm is suitable for every situation. Our findings revealed that simpler models, such as the nearest neighbours classifier, which do not even have a training phase, performed exceptionally well. Additionally, the nearest neighbours classifier offers the desired transparency and interpretability, addressing the reviewer’s concern regarding the opacity of more complex models.

We hope this clarifies our approach and objectives, and we sincerely thank the reviewer for their valuable feedback, which has helped us refine our study and its presentation.

It's not that this reviewer is cynical, but it is fair to suggest a revision conveying a concern for the truly striking lack of organized skepticism in the literature that is being critiqued here. A revision along those lines would serve as a flagship example of exactly the deeper argument that reference (17) was trying to seed, that the applied literature obviously needs a hundred times more of. Such a review would do the most good if it appeared in one of the same journals - AJBA, Evolution, Journal of Human Evolution, Paleobiology - where the bulk of the most highly cited misuses of PCA themselves have appeared.

First, we do not believe that this reviewer is cynical, and we hope they will not consider us cynical if we point out that the field has thus far largely ignored previous reports of PCA misuses published in those journals, like the excellent Bookstein 2019 (4) paper, so perhaps a different approach is needed with a different journal.

Second, our MS is not a review. We agree with the reviewer that a review of PCA critical papers is of value. We changed the title of our study to make it easier to find, and we thank the reviewer for the comment.

Reviewer #3:

Comment 1. Mohseni and Elhaik challenge the widespread use of PCA as an analytical and interpretive tool in the study of geometric morphometrics. The standard approach in geometric morphometrics analysis involves Generalised Procrustes Analysis (GPA) followed by Principal Component Analysis (PCA). Recent research challenges PCA outcomes' accuracy, robustness, and reproducibility in morphometrics analysis. In this paper, the authors demonstrate that PCA is unreliable for such studies. Additionally, they test and compare several Machine-Learning methods and present MORPHIX, a Python package of their making that incorporates the tools necessary to perform morphometrics analysis using ML methods.

Mohseni and Elhaik conducted a set of thorough investigations to test PCA's accuracy, robustness, and reproducibility following renewed recent criticism and publications where this method was abused. Using a set of 2 and 3D morphometric benchmark data, the authors performed a traditional analysis using GPA and PCA, followed by a reanalysis of the data using alternative classifiers and rigorous testing of the different outcomes.

In the current paper, the authors evaluated eight ML methods and compared their classification accuracy to traditional PCA. Additionally, common occurrences in the attempted morphological classification of specimens, such as non-representative partial sampling, missing specimens, and missing landmarks, were simulated, and the performance of PCA vs ML methods was evaluated.

This is a correct description of our MS.

The main problem with this manuscript is that it is three papers rolled into one, and the link doesn't work.

We agree that the manuscript is comprehensive and can probably be broken down into more than one manuscript. However, we do not adhere to the philosophies of the least publishable unit (LPU), the smallest publishable unit (SPU), or the minimum publishable unit (MPU). Instead, we believe in producing high-quality and encompassing studies.

We checked the link thoroughly and ensured it is functional, thank you for your comment.

The title promises a new Python package, but the actual text of the manuscript spends relatively little time on the Python package itself and barely gives any information about the package and what it includes or its usefulness. It is definitely not the focus of the manuscript. The main thrust of the manuscript, which takes up most of the text, is the analysis of the papionin dataset, which shows very convincingly that PCA underperforms in virtually all conditions tested.

We agree. We revised the title to reflect the main issue of the paper. Thank you for your comment.

In addition, the manuscript includes a rather vicious attack against two specific cases of misuse of PCA in paleoanthropological studies, which does not connect with the rest of the manuscript at all.

We consider these case studies of the use of PCA, which resonate with our ultimate goal. First, the previous reviewer suggested that we are beating a “dead horse.” We provide very recent and high-profile test cases to support our position that PCA is a popular and widely used method. Second, we wish to show how researchers use data alternations to cherry-pick results. Third, we focus on one of the use cases (the Homo NS) to demonstrate the poor scientific practices prevalent in this field, such as refusing to share data and breaking Science’s policies to protect this act.

If the manuscript is a criticism of PCA techniques, this should be reflected in the title. If it is a report of a new Python package, it should focus on the package. Otherwise, there should be two separate manuscripts here.

It is a criticism of PCA, and it is now reflected in the title; thank you again.

The criticism of PCA is valid and important. However, pointing out that it is problematic in specific cases and is sometimes misused does not justify labeling tens of thousands of papers as questionable and does not justify vilifying an entire discipline. The authors do not make a convincing enough case that their criticism of the use of PCA in analyzing primate or hominin skulls is relevant to all its myriad uses in morphometrics. The criticism is largely based on statistical power, but it is framed as though it is a criticism of geometric morphometrics in general.

We appreciate the opportunity to address the concerns raised regarding our critique of PCA. The reviewer argues that because we analyzed only primate skulls, we cannot extrapolate that PCA will be biased in analyzing other data (other taxa or other usages). Using the same logic, we can also argue that PCA cannot be used to study NEW taxa and certainly not to detect NOVEL taxa because it was never shown to apply to these taxa. We can further argue that PCA cannot be sued to study ANY taxa since it was never shown to yield correct results (PCA results are justified through circular reasoning and are adjusted when they do not show the desired results). However, that part of our answer is not a defense of our method but rather a further criticism of the field.

To answer the question more directly, our criticism of PCA is rooted in empirical evidence and robust research, including studies by Elhaik (5) and others (6, 7), demonstrating that PCA lacks the power to produce accurate and reliable results. If the reviewer believes that using cats instead of primates will somehow boost the accuracy of PCA, they should, at the very least, explain what morphological properties of cats justify this presumption. Concerning the case of other usages, we clearly noted that “the scope of our study was limited to PCA usage in geometric morphology.” The reviewer did not explain why our analysis is not “convincing enough,” so we cannot address it.

As you know, this issue extends beyond the specific case study of primate or hominin skulls in our research. Despite its widespread use, PCA is heavily relied upon in the field, often without sufficient scrutiny of its limitations. Our intention is not to vilify an entire discipline but to highlight the pervasive and sometimes unquestioning reliance on PCA across many studies in geometric morphometrics. Calling to reevaluate studies based on problematic method is not a vilification, this is by definition science.

While we understand the concern about the generalisability of our findings, our critique is based on the inherent limitations of PCA itself, not merely on statistical power. PCA lacks measurable power, a test of significance, and a null model. Its outcomes are highly sensitive to the input data, making them susceptible to manipulation and interpretation. Moreover, the ability to evaluate various dimensions allows for cherry-picking of results, where different outcomes can be equally acceptable, thus undermining the robustness of conclusions drawn from PCA.

We invite the reviewer to examine the mathematical basis of PCA as demonstrated in Figure 1 of Elhaik (2022) (https://www.nature.com/articles/s41598-022-14395-4/figures/1). We ask the reviewer to explain what in this straightforward calculation—calculating the mean of the dimensions, subtracting the mean from the dimensions, calculating the covariance matrix, and identifying the eigenvalues—convinces them that PCA is suitable for predicting evolutionary relationships between samples. What evidence supports the notion that evolutionary relationships can be inferred by merely subtracting the mean of a matrix? There is none, just as there is no statistical power in this method. PCA does not know what the data mean. It can be applied equally to horse race data and a dataset that records how many times Home Simpsons says his catchphrases. PCA is not an evolutionary method; it’s just a linear transformation. If we ask anyone why they trust it, eventually, we will get the answer that with enough tweaking, PCA results produce what the scientist wants to show, and, most importantly, it will be mathematically accurate (and as mathematically accurate as the result of all possible tweaks). There is nothing specific to hominins about it. If your method produces conflicting results by tweaking the number of samples, species, or landmarks, as we showed, your method is worthless. This is what we demonstrated.

We would also like to note that if we had easier access to more data, we would have extended our analysis further and shown that the bias exists in other species. As explained in our manuscript, we reached out to several scientists who refused to share their data so that we would not show biases in their studies. As this reviewer is undoubtedly aware of the practices in the field, this criticism is extremely unfair.

Finally, arguing that our MS dismisses the entire field of geometric morphometrics is also unfair and provocative. We made no such claim. On the contrary, we offer an unbiased method to replace PCA and improve the accuracy of studies in this field.

We hope this clarifies our position and reinforces the validity of our critique. Thank you for your valuable feedback and for allowing us to address these important points.

Comment 2a. The article's tone is very argumentative and provocative, and non-necessary superlatives and modifiers are used ("...colourful scatterplots", lines 101, 155, 672). While this is an excellent paper and should be studied by morphometrics experts and probably anyone using PCA, the overall tone does nothing to help. It reads somewhat like a Facebook rant rather than a scientific paper (there is still, we hope, a difference between the two). Please tone it down.

Again, we thank the reviewer for considering our work excellent. We regret that the reviewer believes that describing colorful (#101) scatterplots as such is a provocation. We do not feel the same way. “Subsumed” (#155) has been suggested to us by an anonymous reviewer. We changed it to “classified” to satisfy the reviewer (However, Schwartz et al. (2014) raised concerns about the phylogenetic inferences based on PCA results of the geometric morphometrics analysis, noting the failure of the method to capture visually obvious differences between the Dmanisi crania and specimens commonly classified under Homo erectus.). We do not understand the problem with #672, but we revised it to read “However, a growing body of literature criticises the accuracy of various PCA applications, raising concerns about its use in geometric morphometrics.” We hope that this satisfies the reviewer. We made no special effort to be argumentative or provocative. There is no need for that; our results speak for themselves. We did, however, make an effort to communicate the gravity of our findings by citing K. Popper. We do not consider this a provocation.

Comment 2b. The acronym ML is normally used to denote Maximum Likelihood in the context of phylogenetic studies. The authors use it to denote Machine Learning, which many readers may find confusing (this reviewer took a while to realize that it was not referring to Maximum Likelihood). Perhaps leave "machine learning" written in full.

We understand that in some contexts, "ML" typically denotes Maximum Likelihood, which can indeed cause confusion. Unfortunately, “ML” is also a well-established acronym for machine learning, and since our paper doesn’t deal with Maximum Likelihood but rather machine learning, we have to choose the latter. Initially, we did spell out "Machine Learning" in full to avoid this confusion. However, upon review, we found that the manuscript's readability and flow were compromised, leading us to revert to the acronym.

We appreciate your suggestion and understand the importance of clarity. To address this, we will ensure that the first mention of "ML" is accompanied by "Machine Learning" written in full (Line 244). This should help maintain both clarity and readability. Thank you for your valuable input.

Comment 3. In lines 142, 157 Rohlf's should be Rohlf.

(Lines 191, 205) We modified it accordingly and replaced "Rohlf's" with "Rohlf".

Comment 4. The short paragraph in lines 165-167 feels out of place and does not connect to the paragraphs before and after it.

(Lines 210-223) We modified the introduction and merged that paragraph with a relevant paragraph. The new paragraph reads:

“PCA’s prominent role in morphometrics analyses and, more generally, physical anthropology is inconsistent with the recent criticisms, raising concerns regarding its validity and, consequently, the value of the results reported in the literature. To assess PCA’s accuracy, robustness, and reproducibility in geometric morphometric analysis, particularly its potential biases and inconsistencies in clustering with species taxonomy for phylogenetic reconstruction, we utilised a benchmark database containing landmarks from six known species within the Old World monkeys tribe Papionini. We altered this dataset to simulate typical characteristics of paleontological data. We found that PCA’s outcomes lack reliability, robustness, and reproducibility. We also evaluated the argument that a high explained variance could be counted as a measure of reliability (2) and found no association between high explained variance amounts and the subjectiveness of the results. If PCA of morphometric landmark data produces biased results, then landmark-based geometric morphometric studies employing PCA, conservatively estimated to range jfrom 18,400 to 35,200 (as of July 2024) (see Methods), should be reevaluated.”

We thank the reviewer for the suggestion.

References

(1) Gilbert CC, Rossie JB. Congruence of molecules and morphology using a narrow allometric approach. Proceedings of the National Academy of Sciences. 2007;104(29):11910-11914.

(2) Courtenay LA, Yravedra J, Huguet R, Aramendi J, Maté-González MÁ, González-Aguilera D, et al. Combining machine learning algorithms and geometric morphometrics: a study of carnivore tooth marks. Palaeogeography, Palaeoclimatology, Palaeoecology. 2019;522:28-39.

(3) Bellin N, Calzolari M, Callegari E, Bonilauri P, Grisendi A, Dottori M, et al. Geometric morphometrics and machine learning as tools for the identification of sibling mosquito species of the Maculipennis complex (Anopheles). Infection, Genetics and Evolution. 2021;95:105034.

(4) Bookstein FL. Pathologies of between-groups principal components analysis in geometric morphometrics. Evolutionary Biology. 2019;46(4):271-302.

(5) Elhaik E. Principal Component Analyses (PCA)-based findings in population genetic studies are highly biased and must be reevaluated. Scientific reports. 2022;12(1):1-35.

(6) Cardini A, Polly PD. Cross-validated between group PCA scatterplots: a solution to spurious group separation? Evolutionary Biology. 2020;47(1):85-95.

(7) Berner D. Size correction in biology: how reliable are approaches based on (common) principal component analysis? Oecologia. 2011;166(4):961-971.