From salience to semantics: multilevel hierarchical contingencies organise parent-infant joint attention

Reviewer #1 (Public review):

Summary:

This paper investigates infants' social perception as reflected in looking behavior during face-to-face mother-infant toy play in two groups (5 and 15 months). Using information-theoretic and computer-vision methods, the authors quantify dynamic changes in lower-level (salience) and higher-level (semantic) features in the auditory and visual domains - primarily from mothers - and relate these to infants' real-time attention to toys (and to mothers). Time-lagged correlations suggest dynamic, reciprocal relations between infants' attention and maternal low-level (salience) and high-level (semantic) features at both ages, consistent with an early emergence of interpersonal social contingency based on multi-level information during interaction.

Strengths:

The study uses a naturalistic, multimodal mother-infant free-play paradigm and applies information-theoretic/AI methods to quantify both low- and high-level features of maternal behavior, enabling a fine-grained decomposition of interaction dynamics. The time-lag approach further allows examination of temporal relations between maternal signals and infants' attention.

Weaknesses:

Directionality claims from cross-correlations are sometimes unclear, especially when both positive and negative lags are significant, and the evidence for age effects is not yet convincing. Infant attention was manually coded with only moderate-substantial agreement, and handling of disagreements/uncodable periods should be clarified and acknowledged as a limitation.

Reviewer #2 (Public review):

Summary:

This study examines the dynamic interplay between infant attention and hierarchical maternal behaviors from a social information processing perspective. By employing a comprehensive naturalistic framework, the author quantified interactions across both low-level (sensory) and high-level (semantic) features. With correlation analysis with these features, they found that within social contexts, behaviors such as joint attention - shaped by mutual interaction - exhibit patterns distinct from unilateral responding or mimicry. In contrast to traditional semi-structured behavioral observation and coding, the methods employed in this study were designed to consciously and sensitively capture these dynamic features and relate them temporally. This approach contributes to a more integrated understanding of the developmental principles underlying capacities like joint action and communication.

Strengths:

The manuscript's core strength lies in its innovative, dynamic, and hierarchical framework for investigating early social attention. The findings reveal complex adaptive scaffolding strategies: for instance, when infants focus on objects, mothers reduce low-level sensory input, minimising distractions. Furthermore, the results indicate that, even from early development, maternal behaviors are both driven by and predictive of infant attention, confirming that attention involves complex interactive processes that unfold across multiple levels, from salience to semantics.

From a methodological standpoint, the use of unstructured play situations, combined with multi-channel, high-precision time-series analyses, undoubtedly required substantial effort in both data collection and coding. Compared to the relatively two-dimensional analytical approaches common in prior research, this study's introduction of lower-level and higher-level features to explore the hierarchical organization of processing across development is highly plausible. The psychological processes reflected by these quantified physical features span multiple domains - including emotion, motion, and phonetics - and the high temporal sampling rate ensures fine-grained resolution.

Critically, these features are extracted through a suite of advanced machine learning and computational methods, which automate the extraction of objective metrics from audiovisual data. Consequently, the methodological flow significantly enhances data utilization and offers valuable inspiration for future behavioral coding research aiming for high ecological validity.

Weaknesses:

The conclusion of this paper is generally supported by the data and analysis, but some aspects of data analysis need to be clarified and extended.

(1) A more explicit justification for the selection and theoretical categorization of the eight interaction features may be needed. The paper introduces a distinction between "lower-level" and "higher-level" features but does not clearly articulate the criteria underpinning this classification. While a continuum is acknowledged, the practical division requires a principled rationale. For instance, is the classification based on the temporal scale of the features, the degree of cognitive processing required for their integration, or their proximity to sensory input versus semantic meaning?

(2) The claims regarding age-related differences in Predictions 2 are not fully substantiated by the current analyses. The findings primarily rely on observing that an effect is significant in one age group but not the other (e.g., the association between object naming and attention is significant at 15 months but not at 5 months). However, this pattern alone does not constitute evidence about whether the two age groups differ significantly from each other. The absence of a direct statistical comparison (e.g., an interaction test in a model that includes age as a factor) creates an inferential gap. To robustly support developmental change, formal tests of the Age × Feature interaction on infant attention are required.

(3) Another potential methodological issue concerns the potential confounding effect of parents' use of the infant's name. The analysis of "object naming" does not clarify whether utterances containing object words (e.g., "panda") were distinct from those that also incorporated the infant's name (e.g., "Look, Sarah, the panda!"). Given that a child's own name is a highly salient social cue known to robustly capture infant attention, its co-occurrence with object labels could potentially inflate or confound the measured effect of object naming itself. It would be important to know whether and how frequently infants' names were called, whether this variable was analyzed separately, and if its effect was statistically disentangled from that of pure object labeling.

(4) Interpretation of results requires clarification regarding the extended temporal lags reported, specifically the negative correlation between maternal vocal spectral flux and infant attention at 6.54 to 9.52 seconds (Figure 4C). The authors interpret this as a forward-prediction, suggesting that a decrease in acoustic variability leads to increased infant attention several seconds later. However, a lag of such duration seems unusually long for a direct, contingent infant response to a specific vocal feature. Is there existing empirical evidence from infant research to support such a prolonged response latency? Alternatively, could this signal suggest a slower, cyclical pattern of the interaction rather than a direct causal link?

Reviewer #3 (Public review):

Summary:

This manuscript presents an ambitious integration of multiple artificial intelligence technologies to examine social learning in naturalistic mother-infant interactions. The authors aimed to quantify how information flows between mothers and infants across different communicative modalities and timescales, using speech analysis (Whisper), pose detection (MMPose), facial expression recognition, and semantic modeling (GPT-2) in a unified analytical framework. Their goal was to provide unprecedented quantitative precision in measuring behavioral coordination and information transfer patterns during social learning, moving beyond traditional observational coding approaches to examine cross-modal coordination patterns and semantic contingencies in real-time across multiple temporal scales.

Strengths:

The integration of multiple AI tools into a coherent analytical framework represents a genuine methodological breakthrough that advances our capabilities for studying complex social phenomena. The authors successfully analyzed naturalistic interactions at a scale and level of detail that was not previously possible, examining 33 5-month-old and 34 15-month-old dyads across multiple modalities simultaneously. This sophisticated analytical pipeline, combining speech analysis, semantic modeling, pose detection, and facial expression recognition, provides new capabilities for studying social interactions that extend far beyond what traditional observational coding could achieve.

The specific findings about hierarchical information flow patterns across different timescales are particularly valuable and would not have been possible without this sophisticated analytical approach. The discovery that mothers reduce low-level sensory input when infants focus on objects, while increases in object naming and information rate associate with sustained attention, provides new empirical insights into how social learning unfolds in naturalistic settings. The temporal dynamics analyses reveal interesting patterns of behavioral coordination that extend our understanding of how caregivers adaptively modify their responses to support infant attention across multiple communicative channels simultaneously.

The scale of data collection and the comprehensive multi-modal approach are impressive, opening up new possibilities for understanding social learning processes. The methodological innovations demonstrate how modern computational tools can be systematically integrated to reveal new quantitative aspects of well-established developmental phenomena. The computational features developed for this study represent innovative applications of information theory and computer vision to developmental research.

Weaknesses:

Several major limitations affect the reliability and interpretability of the findings. The sample sizes of 33-34 dyads per age group are relatively modest for the complexity of analyses performed, which include eight different features examined across various time lags with extensive statistical comparisons. The study lacks adequate power analysis to demonstrate whether these sample sizes are sufficient to detect meaningful effect sizes, which is particularly concerning given the multiple comparison burden inherent in this type of multi-modal, multi-timescale analysis.

The statistical framework presents several concerns that limit confidence in the findings. Inter-rater reliability for gaze coding shows substantial but not excellent agreement (κ = 0.628), with only 22% of the data undergoing double coding. Given that gaze coding forms the foundation for all subsequent analyses of joint attention and information flow, this reliability level may systematically influence findings. The multiple comparison correction strategies vary inconsistently across different analyses, with some using FDR correction and others treating lower-level and higher-level features separately. Additionally, object naming analyses employed one-sided tests (p<0.05) while others used two-sided tests (p<0.025) without clear theoretical or methodological justification for these differences.

The validation of AI tools in the specific context of mother-infant interactions is insufficient and represents a critical limitation. The performance characteristics of Whisper with infant-directed speech, the precision of MMPose for detecting facial landmarks in young children, and the accuracy of facial expression recognition tools in infant contexts are not adequately validated for this population. These sophisticated tools may not perform optimally in the specific context of mother-infant interactions, where speech patterns, facial expressions, and body movements may differ substantially from their training data.

The theoretical positioning requires substantial refinement to better acknowledge the extensive existing literature. The authors are working within a well-established theoretical framework that has long recognized social learning as an active, bidirectional process. The joint attention literature, beginning with foundational work by Bruner (1983) and continuing through contemporary theories of social cognition by researchers like Tomasello (1995), has emphasized the communicative and adaptive nature of attentional processes. The scaffolding literature, including seminal work by Wood, Bruner, and Ross (1976), has demonstrated how parents adjust their support based on children's developing competencies. Moreover, there is a substantial body of micro-analytic research that has employed sophisticated quantitative methods to study social interactions, including work by Stern (1985) on microsecond-level interactions and research using time-series methods to examine dyadic coordination patterns.

The cross-correlation analyses have inherent limitations for causal inference that are not adequately acknowledged. The interpretation of temporal correlation patterns in terms of directional influence requires more cautious consideration, as observational data have fundamental constraints for establishing causality. The ecological validity is also questionable due to the laboratory tabletop interaction paradigm and the sample's demographic homogeneity, consisting primarily of white, highly educated, high-income mothers.