A Monte Carlo EM approach for partially observable diffusion processes: theory and applications to neural networks.

Full Abstract

We present a Monte Carlo approach for training partially observable diffusion processes. We apply the approach to diffusion networks, a stochastic version of continuous recurrent neural networks. The approach is aimed at learning probability distributions of continuous paths, not just expected values. Interestingly, the relevant activation statistics used by the learning rule presented here are inner products in the Hilbert space of square integrable functions. These inner products can be computed using Hebbian operations and do not require backpropagation of error signals. Moreover, standard kernel methods could potentially be applied to compute such inner products. We propose that the main reason that recurrent neural networks have not worked well in engineering applications (e.g., speech recognition) is that they implicitly rely on a very simplistic likelihood model. The diffusion network approach proposed here is much richer and may open new avenues for applications of recurrent neural networks. We present some analysis and simulations to support this view. Very encouraging results were obtained on a visual speech recognition task in which neural networks outperformed hidden Markov models.

Learn Faster Today Improve your study skills

Author information

Author/s: Movellan, Javier R (JR); Mineiro, Paul (P); Williams, R J (RJ);

Affiliation: Machine Perception Laboratory, Institute for Neural Computation, University of California, San Diego, La Jolla, CA 92093, USA. movellan(-atsign-)inc.ucsd.edu

Journal and publication information

Publication Type: Journal Article; Research Support, U.S. Gov't, Non-P.H.S.

Journal: Neural computation (Neural Comput), published in United States. (Language: eng)

Reference: 2002-Jul; vol 14 (issue 7) : pp 1507-44

Dates: Created 2002/06/24; Completed 2002/07/16; Revised 2006/11/15;

PMID: 12079544, status: MEDLINE (last retrieval date: 11/6/2008)

Sourced from the National Library of Medicine. Abstract text and other information may be subject to copyright.

External Links for this article (including full text providers, if available):

Click Electronic Full-text Provider Links to see options for finding the electronic full text links to this article. Note there may be a subscription or fee required for access to the full text. See our FAQ for information on finding FREE full text articles.

This article may also be located in paper journal collections available in many libraries. Use the Journal and Publication Information above to find the full article.

MeSH headings (categories)

This article was linked to the MESH Headings shown below.

This article has not been indexed for related articles as yet, however you can still use the live related article search links below.

See 100+ related articles.