The first day began with an introductory phase and a tour around the Somabotics lab by Assistant Professor Juan Ávila and PhD candidate Marco Amerotti. During this first encounter, we wanted to draw parallels between the research being done at the lab, specifically the work in the ‘Traditionable Machines’ project. Marco presented LOERIC, a symbolic AI music system designed for live improvisation around folk Irish music, followed by a demo. The day continued with a presentation of my AI-powered music system  SEMILLA.AI, which is a generative AI, neural synthesis instrument with an interface inspired by the ancient divination practice of ‘seed casting’. We then mapped out the differences between the LOERIC approach (symbolic AI) and SEMILLA.AI (generative AI), exploring potential ways to combine and create a neuro-symbolic system. One of the insights we arrived at was that that Symbolic AI systems can be quite resilient in very narrow domain genres, and are customizable for particular compositions, as opposed to SEMILLA.AI, which acts as a timbral and sound synthesis engine, where the musician must engage in an active listening exercise in order to steer the system in musically meaningful directions. LOERIC engages with pre-written rules and fuzzy logic to be flexible for co-improvisation, while SEMILLA.AI is all about active listening, performance and exploration of an unknown, model dependent and high dimensional timbral space.

The visit continued into the second day, where Juan, Marco and I set up a small, hacklab style space to see how to combine both LOERIC and SEMILLA.AI.  We developed a prototype where the latent parameters of SEMILLA.AI were controlled by certain confidence parameters from the LOERIC system, which in itself was taking a live audio input. We had a jam session and a subsequent discussion on the ‘feel’ of the system. We arrived at the conclusion that it might be more productive to design a hybrid music system from the ground up (symbolic and neural), as the implicit biases present in the design decision of LOERIC and SEMILLA.AI grounded both instruments within these paradigms, respectively.

On the third day of my visit I was introduced to the Somabotics Somaskins research team and participated in a prototyping session with this framework. I was introduced also to the work being done within the field of soft robotics, and certain projects and prototypes being developed at the lab. Guided by the developers, I had the chance to experiment with the Somaskins kit, which includes pneumatic activated interfaces, and was introduced to the different approaches to this interface. I am really interested in developing a hybrid approach about how to both embody AI systems, but also haptically ‘feel’ the latent space (or internal hidden representations) of black box generative AI music systems.

After being introduced to some of the prototyping kits for soft-robotics and Somaskins, we moved into conceptualizing a prototype of a soft-robotics augmented musical instrument, particularly thinking of ways to create an augmented instrument with a ceramic self-built ocarina. As one of the central themes of the Somabotics Fellowship is to develop embodied approaches to AI, I am particularly interested in the affordances that soft robotics, pneumatics, machine listening and feedback systems can bring to develop a new music instrument. Day four continued with the use of a soft robotic system, which was a ‘co-improviser’, using live audio input – RAVE – for both encoding the audio into latent code representations and decoding back into audio. I developed an early prototype that proved to be interesting – an active listening instrument, which from its hidden representations (latent codes) of the audio input, engages in embodied behaviours which generate melodies by physically pressing on the ceramic ocarina, enabling a human-embodied AI improvisation. The prototype will be the starting point for a residency which aligns with the embodied AI and Traditionable Machines project. I will develop an augmented musical instrument/sculpture, which can inhabit the room, is interactive, but also automatically reflects the AI’s internal states using soft robotics and pneumatics as the interface for interaction and embodiment.

My visit came to an  end with an lecture that I delivered to members of the School of  Computer Science at the University of Nottingham along with Generative AI specialists. My work was framed within an artist-centered approach to generative AI, engaging critically with new technologies, while developing protocols for artist-first approaches to ethics in AI music and music technology development.

This visit was an introduction to the Somabotics areas of research and a stepping stone towards a residency that will involve collaboration, development, creation and exploration into the roles that soft-robotics, the Somaskins kit and AI music can have as a way to rethink musical interfaces for AI music systems. I’m particularly interested in developing more strategies which explore the affordances that an embodied approach to generative AI music systems can have, both on the user and spectator. The aim will be to co-develop a larger scale musical instrument which will be presented as both a sound installation and a playable AI music instrument. Many thanks to Professor Steve Benford, Dr. Juan Ávila, Marco Amerotti and the rest of the Somabotics team for their kind insights, exchange of ideas and feedback on where to take the next phase of this work.