A flow chart of conventional modern geometric morphometric analysis. The standard pipeline comprises several steps (left). First, a set of landmarks are chosen based on distinct anatomical features believed to be equivalent in some sense (e.g., biologically homologous) among all the specimens (papionins), and their cartesian coordinates are scanned and recorded. Next, the landmark data are superimposed using GPA through translation, rotation, and scaling. Next, after performing PCA, the scatterplots of the first two or three PCs are used to assess patterns of shape variation based on the distribution of the observations. The outcome of the pipeline (right) are depicted for the benchmark data. For brevity, only three taxa are shown on the top right.

The three papionins and their skull pictures presented in Figure 1 were obtained from the following links under a Creative Commons Attribution-Share Alike 3.0 Unported license with the exception of the last link, which was used with permission: https://tinyurl.com/mrh3ukz8, https://tinyurl.com/2p8777j4, https://tinyurl.com/7dxbt4zb, https://tinyurl.com/ar4uujdj, https://tinyurl.com/47n9z7ae, https://tinyurl.com/mxvr439c. All the pictures were cropped. For two skulls, the background was removed.

PC scatterplots of the shape space of the lower molar of Homo NR2 and other Hominins (left). The analysis combines the enamel–dentine junction (EDJ) and the cemento-enamel junction (CEJ). The Kernel Density Estimation (KDE) plots the distribution of the distances of each Homo species from Homo NR in the space of the corresponding PCs (right). The legends of the right sub-figures present the adjusted (Benjamini/Hochberg) p-values of the T-test for the means of the distances of each Hominin population and Neanderthals from Homo NR. Legend guide: African Early Homo sapiens (AEHs), African Middle Pleistocene Homo (MPH), Early Homo sapiens (EHs), European Middle Pleistocene Homo (EMPH), Homo erectus (He), Recent Homo sapiens (RHs). For consistency, the colours of the species are similar to those of Hershkovitz et al. (53) PC plots.

The PC scores use for creating Figure 2 were obtained from Israel Hershkovitz.

Plots depicting the papionin benchmark data. A) PC 1-2, B) PC 1-3, C) PC 2-3, D) Confusion matrix of a 2NN classifier with 5-fold CV, E) Dendrogram of Procrustes distances of the samples using agglomerative clustering, F) Decision boundary of the 2NN classifier after t-SNE embedding with 5-fold CV.

The accuracy and balanced accuracy of the eight classifiers. The accuracy was evaluated in a five-fold stratified CV on the benchmark dataset and the six alteration cases. The numbers in parentheses are the balanced accuracies.

Plots depicting the papionin data after the removal of the Lophocebus albigena (green) taxon. A) PC 1-2, B) PC 1-3, C) PC 2-3, D) 2-D t-SNE scatterplot, E) Dendrogram of Procrustes distances of the samples using agglomerative clustering, F) Decision boundary of the 2NN classifier after t-SNE embedding with 5-fold CV. The insets in the subfigures are the corresponding benchmark PC scatterplots (Figure 3A-C).

Plots depicting the papionin data after removing the Macaca mulatta (orange) taxon. A) PC 1-2, B) PC 1-3, C) PC 2-3, D) 2-D t-SNE scatterplot, E) Dendrogram of Procrustes distances of the samples using agglomerative clustering, F) Decision boundary of the 2NN classifier after t-SNE embedding with 5-fold CV. The insets in the subfigures are the corresponding benchmark PC scatterplots (Figure 3A-C).

Plots depicting the papionin data after the first case of sample removal. 11 Lophocebus albigena (green) samples (index in benchmark dataset: 52, 56, 60, 63, 64, 66, 67, 73, 75, 76 and 77). A) PC 1-2, B) PC 1-3, C) PC 2-3, D) 2-D t-SNE scatterplot, E) Decision boundary of 2NN classifier after t-SNE embedding, F) Decision boundary with 5-fold CV. The insets in the subfigures are the corresponding benchmark PC scatterplots (Figure 3A-C). The Lophocebus albigena (pink) sample is marked by a red arrow.

Plots depicting the papionin data after the second case of sample removal. 19 samples from both Lophocebus albigena (green) (index in benchmark dataset: 52, 56, 60, 63, 64, 66, 67, 73, 75, 76 and 77) and Cercocebus torquatus (blue) (index in benchmark dataset: 80, 81, 82, 83, 84, 88, 91 and 94) A) PC 1-2, B) PC 1-3, C) PC 2-3, D) 2-D t-SNE scatterplot, E) Decision boundary of 2NN classifier after t-SNE embedding, F) Decision boundary with 5-fold CV. The insets in the subfigures are the corresponding benchmark PC scatterplots (Figure 3A-C).

Plots depicting the papionin data after the first case of landmark removal. Landmarks 23, 24, 25, 26 and 27 were removed. Scatterplots of the papionin data after the first case of sample removal. A) PC 1-2, B) PC 1-3, C) PC 2-3, D) 2-D t-SNE scatterplot, E) Decision boundary of 2NN classifier after t-SNE embedding, F) Decision boundary with 5-fold CV. The insets in the subfigures are the corresponding benchmark PC scatterplots (Figure 3A-C).

Plots depicting the papionin data after the second case of landmark removal. Landmarks 11, 12, 16, 17, 18, 30 (symmetrical pair of 12) and 31 (pair of 13) were removed. Scatterplots of the papionin data after the first case of sample removal. A) PC 1-2, B) PC 1-3, C) PC 2-3, D) 2-D t-SNE scatterplot E) Decision boundary of 2NN classifier after t-SNE embedding, F) Decision boundary with 5-fold CV. The insets in the subfigures are the corresponding benchmark PC scatterplots (Figure 3A-C).

Plots depicting the papionin data in the first extreme case. A) PC 1-2, B) PC 1-3, C) PC 2-3, D) 2-D t-SNE scatterplot E) Decision boundary of 2NN classifier after t-SNE embedding, F) Decision boundary with 5-fold CV. The insets in the subfigures are the corresponding benchmark PC scatterplots (Figure 3A-C). the corresponding benchmark PC scatterplots (Figure 3A-C).

Plots depicting the papionin data in the second extreme case. A) PC 1-2, B) PC 1-3, C) PC 2-3, D) 2-D t-SNE scatterplot E) Decision boundary of 2NN classifier after t-SNE embedding, F) Decision boundary with 5-fold CV. The insets in the subfigures are the corresponding benchmark PC scatterplots (Figure 3A-C).

Detecting outliers using local outlier score and KDE plots. Each Cercocebus torquatus (blue) sample is treated as an outlier (red arrow, A1-O1 t-SNE plots) and the KDE of the scores are plotted for each case (A2-O2). The radius of the circle around each sample is proportional to the LOF score. The red dot in the KDE plots shows the LOF score of the outlier in each case.

Detecting outliers using local outlier score and KDE plots. Six Cercocebus torquatus (blue) samples (indexes: 84,85,86,87,90 and 92) are treated as outliers (A1 t-SNE plots) and the KDE of the scores are plotted (A2). The radius of the circle around each sample is proportional to the LOF score. The red dots in the KDE plots show the LOF score of the outliers.

Detecting outliers using local outlier score and KDE plots. All Cercocebus torquatus (blue) samples are treated as outliers (A1 t-SNE plot) and the KDE of the scores are plotted (A2). The radius of the circle around each sample is proportional to the LOF score. The red dots in the KDE plots show the LOF score of the outliers.

The LOF outlier scores. Cercocebus torquatus (blue) was treated as an outlier in three cases; A) each sample separately, B) 6 samples together, and C) the whole group together.

The 31 landmarks of Table S1. The wireframes connect the landmarks.