This piece explores audiovisual composition using hand gestures as the primary means of interaction. It adapts the Sabetan technique of Indonesia's Wayang Kulit shadow puppetry, aiming to demonstrate embodied audio-visual interaction. Sabetan involves expressive gestures by the puppeteer, conforming to the scene's characterization.
The embodied interaction is performative, creating audio and visual elements simultaneously. It draws on Michael Chion's "added-value" concept. A choreographed hand movement, incorporating Sabetan, facilitates real-time interaction with a depth camera and machine learning. This concept is implemented in Ciung Wanara, an interactive music system integrating Wayang Kulit and supervised machine learning.
The system also employs the Polymorphism Sound SYNTHESIS method. "Ciung Wanara" is an interactive audiovisual composition inspired by Indonesian shadow puppet battles. It features rooster avatars responding to the performer's gestures, recreating the tale of Ciung Wanara, a wise king defeating an evil sorcerer. The Sabetan technique and FluComa machine learning enable simultaneous control of sound synthesis, spatialization, and visuals. Two avatars with a narrative driven by hand gestures create a unique immersive experience, uncommon in traditional audiovisual composition, engaging both performer and audience.