The Music, Audio, and AI Lab (AMAAI) at SUTD invites applications for a PhD position in the exciting and rapidly evolving field of music and audio artificial intelligence.
The AMAAI lab is engaged in cutting-edge research at the intersection of music, audio, and artificial intelligence. Our PhD students contribute to groundbreaking projects that explore areas such as:
I am excited to announce the latest paper by Viet-Toan Le, who was a visiting student at the AMAAI Lab, on 'Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey'. Viet-Toan did an amazing job at collating and nicely presenting over 225 papers in Music Information Retrieval that are inspired by NLP, as well as presenting the latest challenges and steps forward for the field!
I am excited to announce that we have two positions open at the AMAAI Lab in Singapore: a postdoc and research assistant position in generative AI for music models. They will be building on our recent work of the Mustango model (https://github.com/AMAAI-Lab/mustango) and video2music (https://github.com/AMAAI-Lab/Video2Music).
Excited to announce Mustango, a powerful multimodal Model for generating music from textual prompts. Mustango leverages a Latent Diffusion Model conditioned on textual prompts (encoded using Flan-T5) and various musical features. Try the demo! What makes it different from the rest?
-- greater controllability in the music generation.
-- trained on a large dataset generated using ChatGPT and musical manipulations.
-- superior performance over its predecessors as per the experts.
-- open source!
We are happy to announce Video2Music, a novel AI-powered multimodal music generation framework called Video2Music. This framework uniquely uses video features as conditioning input to generate matching music using a Transformer architecture. By employing cutting-edge technology, our system aims to provide video creators with a seamless and efficient solution for generating tailor-made background music.
We are happy to announce two talks on Tuesday 17 October at 2pm at SUTD I3 Lab 1.605
Title : Exploring NLP Methods in Symbolic MIR: Representations and Models
Our paper on PreBit - A multimodal model with Twitter FinBERT embeddings for extreme price movement prediction of Bitcoin just got published in Expert Systems with Applications.
Congratulations to Joel Ong on publishing our paper on using multi-task deep learning for porfolio construction in Expert Systems with Applications. The paper presents a new way to leverage time series momentum in a deep learning setting. Read a Twitter thread explaining the basics here.
Great work Cheuk Kin Wai on his latest paper on DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability
Cheuk, K. W., Sawata, R., Uesaka, T., Murata, N., Takahashi, N., Takahashi, S., ... & Mitsufuji, Y. (2022). DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability. arXiv preprint arXiv:2210.05148.
Congratulations to Thao on leading the publication of the EmoMV dataset set for music-video matching based on emotion!
Pham Q-H, Herremans D., Roig G.. 2022. EmoMV: Affective Music-Video Correspondence Learning Datasets for Classification and Retrieval. Information Fusion. DOI: 10.1016/j.inffus.2022.10.002
Paper highlights:
I was honoured to give a keynote talk at the 3rd Conference on AI Music Creativity (AIMC) on controllable music generation with emotion. Watch the full keynote here:
Congrats on my former research assistant Jiahui Huang on his latest paper in Sensors on 'Single Image Video Prediction with Auto-Regressive GANs'. Now we can generate videos of faces with desired emotions!
Huang, Jiahui, Yew Ken Chia, Samson Yu, Kevin Yee, Dennis Küster, Eva G. Krumhuber, Dorien Herremans, and Gemma Roig. "Single Image Video Prediction with Auto-Regressive GANs." Sensors 22, no. 9 (2022): 3533.
Read my PhD student Yanzhao Zou and myself's latest paper on A Multimodal Model with Twitter Finbert Embeddings for Extreme Price Movement Prediction of Bitcoin.
SUTD Game Lab is looking for a passionate game developer to join our team. The Game Programmer will visualise how systems should work, and translate this into a functioning solution using the existing game engine to develop prototypes or a polished game for training purposes.
Responsibilities:
Read the full NRF magazine here. Our original paper is available here.
It was an honour today to be part of the seminar at the KTH Royal Institute of Technology in Stockholm as part of the dialogues series.
dialogues1: probing the future of creative technology
Subject: “Interaction with generative music frameworks”
Guests: Dorien Herremans and Kıvanç Tatar (Video link to be posted)
Dorien Herremans: Controllable deep music generation with emotion
Excited to be featured on the latest 'AI and You - What is AI? How will it affect your life, your work, and your world?' podcast by Peter Scott from Human Cusp.
We're focusing on AI in music: What's the state of the art in AI music composition, how can human composers use it to their advantage, and what is the AI Song Contest? How do musical AIs surprise their creators and how are they like your grandmother trying to explain death metal?
Last year I was honoured to be part of the panel discussion on 'Challenging the limits of AI for the next generation of co-creative tools - Frontiers of Music and Artificial Intelligence'. at Ars Electronica, IRCAM (FR). Watch the video below.
Congrats to Kin Wa Cheuk for his published paper in the ACM Multimedia conference (A*) on 'ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data'. If you are interested in training low-data music transcription models with semi-supervised learning, check out the full paper here, or access the preprint.
Watch Raven's talk here:
Sounder Music in the Netherlands (https://soundersmusic.com/) has an internship opportunity for a MSc or PhD student in data analytics for music. The internship will be (remotely) co-supervised by myself (Prof. Dorien Herremans, SUTD) and the founder of Sounders Music (Willem Bloem).
