Genetic variation is known to contribute to the variation of animal social behavior, but the molecular mechanisms that lead to behavioral differences are still not fully understood. Here, we investigate the cellular evolution of the hypothalamic preoptic area (POA), a brain region that plays a critical role in social behavior, across two sister species of deer mice (Peromyscus maniculatus and P. polionotus) with divergent social systems. These two species exhibit large differences in mating and parental care behavior across species and sex. Using single-nucleus RNA-sequencing, we build a cellular atlas of the POA for males and females of both Peromyscus species. We identify four cell types that are differentially abundant across species, two of which may account for species differences in parental care behavior based on known functions of these cell types. Our data further implicate two sex-biased cell types to be important for the evolution of sex-specific behavior. Finally, we show a remarkable reduction of sex-biased gene expression in P. polionotus, a monogamous species that also exhibits reduced sexual dimorphism in parental care behavior. Our POA atlas is a powerful resource to investigate how molecular neuronal traits may be evolving to give rise to innate differences in social behavior across animal species.