Research in generative self-supervised learning (SSL) has largely focused on local embeddings for tokenized sequences. We introduce a generative SSL framework that learns a global representation that is disentangled from local embeddings. We apply this technique to jointly learn a global speaker embedding and a zero-shot voice converter. The converter modifies recorded speech to sound as if it were spoken by a different person while preserving the content, using only a short reference clip unavailable to the model during training. Listening experiments conducted on an unseen dataset show that our models significantly outperform SOTA baselines in both quality and speaker similarity for various datasets and unseen languages.

@inproceedings{Wang:2024:GSG,
   author = "Yunyun Wang and Jiaqi Su and Adam Finkelstein and Zeyu Jin",
   title = "{GR0}: Self-supervised Global Representation Learning for Zero-shot
      Voice Conversion",
   booktitle = "ICASSP 2024, to appear",
   year = "2024",
   month = apr
}