Generative power of a protein language model trained on multiple sequence alignments

Abstract
Data availability
Article and author information
Metrics

Abstract

Computational models starting from large ensembles of evolutionarily related protein sequences capture a representation of protein families and learn constraints associated to protein structure and function. They thus open the possibility for generating novel sequences belonging to protein families. Protein language models trained on multiple sequence alignments, such as MSA Transformer, are highly attractive candidates to this end. We propose and test an iterative method that directly employs the masked language modeling objective to generate sequences using MSA Transformer. We demonstrate that the resulting sequences score as well as natural sequences, for homology, coevolution and structure-based measures. For large protein families, our synthetic sequences have similar or better properties compared to sequences generated by Potts models, including experimentally-validated ones. Moreover, for small protein families, our generation method based on MSA Transformer outperforms Potts models. Our method also more accurately reproduces the higher-order statistics and the distribution of sequences in sequence space of natural data than Potts models. MSA Transformer is thus a strong candidate for protein sequence generation and protein design.

Data availability

Python code for generating sequences using the iterative masking procedure is available in our GitHub repository: https://github.com/Bitbol-Lab/Iterative_masking.Raw data was collected from two public sources: 1) MSAs from the Pfam database (https://pfam.xfam.org/); 2) further MSAs from https://github.com/matteofigliuzzi/bmDCA. We generated sequences with bmDCA using code publicly available at https://github.com/ranganathanlab/bmDCA.

Article and author information

Author details

Damiano Sgarbossa

Institute of Bioengineering, Ecole Polytechnique Federale de Lausanne (EPFL), Lausanne, Switzerland

Competing interests
The authors declare that no competing interests exist.

"This ORCID iD identifies the author of this article:" 0000-0002-7878-6061
Umberto Lupo

Institute of Bioengineering, Ecole Polytechnique Federale de Lausanne (EPFL), Lausanne, Switzerland

For correspondence
umberto.lupo@epfl.ch

Competing interests
The authors declare that no competing interests exist.
Anne-Florence Bitbol

Institute of Bioengineering, Ecole Polytechnique Federale de Lausanne (EPFL), Lausanne, Switzerland

For correspondence
anne-florence.bitbol@epfl.ch

Competing interests
The authors declare that no competing interests exist.

"This ORCID iD identifies the author of this article:" 0000-0003-1020-494X

Funding

European Research Council (851173)

Damiano Sgarbossa
Umberto Lupo
Anne-Florence Bitbol

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Copyright

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.