Listening to Uncertainty: A Fuzzy Logic Approach to Visual-to-Audio Generation

Abstract

Cross-modal synthesis from silent video to audio is challenging because visual events do not determine a unique sound and the relationship between visual and acoustic representations is uncertain. This paper presents a Neuro-Fuzzy Visual-to-Audio (NF-V2A) framework that combines a pre-trained 3D Convolutional Neural Network (3D-CNN) for spatial-temporal visual feature extraction, an adaptive Takagi-Sugeno-Kang (TSK) Fuzzy Inference System (FIS) for uncertainty-aware mapping, and a generative adversarial network (GAN) decoder for Mel-spectrogram synthesis. A Genetic Algorithm (GA) optimizes fuzzy membership parameters and rule weights using a fitness objective based on spectrogram Mean Squared Error (MSE) and Fréchet Audio Distance (FAD). Experiments are reported on VGG-Sound and a curated AudioSet subset. On the VGG-Sound test set, the proposed model reports FAD of 3.04, Inception Score of 7.41, KL divergence of 1.12, and spectrogram MSE of 0.043. Compared with the Diff-V2A baseline, the reported FAD reduction is approximately 14.3%, while spectrogram MSE is reduced by approximately 15.6%. The results suggest that combining fuzzy inference and evolutionary optimization can improve the reported perceptual and reconstruction measures for visual-to-audio generation.