Liming Wang(University of Illinois, Urbana Champaign) and Mark Hasegawa-Johnson(University of Illinois)
Abstract:
Discovering word-like units without textual transcriptions is an
important step in low-resource speech technology. In this work,
we demonstrate a model inspired by statistical machine translation
and hidden Markov model/deep neural network (HMMDNN)
hybrid systems. Our learning algorithm is capable of discovering
the visual and acoustic correlates of K distinct words
in an unknown language by simultaneously learning the mapping
from image regions to concepts (the first DNN), the mapping
from acoustic feature vectors to phones (the second DNN),
and the optimum alignment between the two (the HMM). In the
simulated low-resource setting using MSCOCO and Speech-
COCO datasets, our model achieves 62.4 % alignment accuracy
and outperforms the audio-only segmental embedded GMM approach
on standard word discovery evaluation metrics.