Example-Guided Physically Based Modal Sound Synthesis

Zhimin Ren, Hengchin Yeh, and Ming C. Lin

University of North Carolina at Chapel Hill


Linear modal synthesis methods have often been used to generate sounds for rigid bodies. One of the key challenges in widely adopting such techniques is the lack of automatic determination of satisfactory material parameters that recreate realistic audio quality of sounding materials. We introduce a novel method using pre-recorded audio clips to estimate material parameters that capture the inherent quality of recorded sounding materials. Our method extracts perceptually salient features from audio examples. Based on psychoacoustic principles, we design a parameter estimation algorithm using an optimization framework and these salient features to guide the search of the best material parameters for modal synthesis. We also present a method that compensates for the differences between the real-world recording and sound synthesized using solely linear modal synthesis models to create the final synthesized audio. The resulting audio generated from this sound synthesis pipeline well preserves the same sense of material as a recorded audio example. Moreover, both the estimated material parameters and the residual compensation naturally transfer to virtual objects of different sizes and shapes, while the synthesized sounds vary accordingly. A perceptual study shows the results of this system compares well with real-world recordings in terms of material perception.


Example-Guided Physically Based Modal Sound Synthesis

ACM Transactions on Graphics (TOG), SIGGRAPH 2013
Volume 32 Issue 1, January 2013 Article No. 1

Zhimin Ren, Hengchin Yeh, and Ming C. Lin

Preprint PDF
Appendix PDF

Auditory Perception of Geometry-Invariant Material Properties

IEEE Transactions on Visualization and Computer Graphics (TVCG)
IEEE VR 2013 (Honorable Mention)
Volume 19, Issue 4, April 2013, pp. 557-566

Zhimin Ren, Hengchin Yeh, Roberta Klatzky, and Ming C. Lin

Preprint PDF

AudioCloning: Extracting Material Fingerprints from Example Audio Recording

SIGGRAPH 2012 Studio Talk

Hengchin Yeh, Zhimin Ren, and Ming C. Lin

Paper PDF
Supplementary Material


Video download