Unified Semantic Representation of Music Scores, Lyrics, and Singing Voices for Multimodal Artificial Intelligence
Keywords:
unified semantic representation; multimodal artificial intelligence; music information retrieval; singing voice synthesis; score-lyrics alignment; data governanceAbstract
The computational study of music has long been fragmented across symbolic scores, textual lyrics, and audio recordings of singing voices. This fragmentation limits the capacity of artificial intelligence systems to reason across musical modalities, to align lyrical semantics with melodic structure, and to generate expressive performances from combined symbolic and linguistic intent. This paper examines the problem of constructing unified semantic representations for music scores, lyrics, and singing voices within multimodal artificial intelligence. Rather than proposing a single model, the paper develops a system-level account of the structural trade-offs involved in representing discrete symbolic events, continuous acoustic signals, and natural language under a shared semantic framework. The analysis addresses architectural strategies based on attention mechanisms, self-supervised audio representations, and sequence-to-sequence modeling, and it situates these strategies within data infrastructure, annotation governance, and deployment contexts. Particular attention is given to alignment granularity, temporal hierarchy, modality-specific fidelity, robustness across genres and languages, fairness, sustainability, and policy implications. The paper argues that meaningful progress requires not only technical integration but also careful governance of training corpora, evaluation protocols, and downstream sociotechnical effects. A unified representation should be understood as an evolving infrastructure rather than a static embedding space.
References
1. Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798–1828.
2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
3. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171–4186.
4. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460.
5. Huang, C. Z. A., Vaswani, A., Uszkoreit, J., Simon, I., Hawthorne, C., Shazeer, N., ... & Eck, D. (2019). Music Transformer: Generating music with long-term structure. International Conference on Learning Representations.
6. Zhang, Y., Pan, C., Guo, W., Li, R., Zhu, Z., Wang, J., ... & Zhao, Z. (2024). Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37, 1117-1140.
7. Choi, K., Fazekas, G., Sandler, M., & Cho, K. (2017). Convolutional recurrent neural networks for music classification. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2392–2396.
8. van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). Neural discrete representation learning. Advances in Neural Information Processing Systems, 30.
9. Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., ... & Frank, C. (2023). MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325.
10. Oore, S., Simon, I., Dieleman, S., Eck, D., & Simonyan, K. (2020). This time with feeling: Learning expressive musical performance. Neural Computing and Applications, 32, 955–967.
11. Müller, M. (2015). Fundamentals of music processing: Audio, analysis, algorithms, applications. Springer.
12. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of the 40th International Conference on Machine Learning, 28492–28518.
13. Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T. Y. (2021). FastSpeech 2: Fast and high-quality end-to-end text-to-speech. International Conference on Learning Representations.
14. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru, T. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220–229.
15. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92.
16. Schedl, M., Gómez, E., & Urbano, J. (2014). Music information retrieval: Recent developments and applications. Foundations and Trends in Information Retrieval, 8(2-3), 127–261.
17. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623.
18. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–3650.
19. Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green AI. Communications of the ACM, 63(12), 54–63.
20. Zeng, M., Tan, X., Wang, R., Ju, D., Qin, T., & Liu, T. Y. (2021). MusicBERT: Symbolic music understanding with large-scale pre-training. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 791–800.
21. Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., ... & Bengio, Y. (2021). Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research, 22(1), 7459–7478.
22. Chesney, R., & Citron, D. K. (2019). Deep fakes: A looming challenge for privacy, democracy, and national security. California Law Review, 107, 1753–1820.
23. Veale, M., & Zuiderveen Borgesius, F. (2022). Demystifying the draft EU artificial intelligence act—analysing the good, the bad, and the unclear elements of the proposed approach. Digital Society, 1, 1–24.
24. Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., ... & Mons, B. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3, Article 160018.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Artificial Intelligence and Machine Learning Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.