[Topic-models] samplers for LDA likelihood

Wray Buntine wray.buntine at nicta.com.au
Mon Mar 29 21:01:10 EDT 2010

In response to David Mimno's questions:

>  is there a general way to define an optimal mean field 
>  importance sampler for a new topic model? What would the mean-field 
>  importance sampler for CTM look like, for example?

Well, one needs to develop the mean field approximation.
Other than using Ghahramani and Beal's elegant formulation
("Propagation Algorithms for Variational {B}ayesian Learning", 2000),
which makes mean-field simple, I cannot really say much.

We've adopted the "left-to-right sequential sampler" for some
other topic models, but not the mean-field importance
sampler.  I expect the mean-field importance sampler could
have more general uses.

>  Also, what is the relationship between this mean field approximation and 
>  the standard variational approximation used in training models? The update 
>  in Eq. 4 doesn't quite match the standard variational update, for example, 
>  which seems like it should also minimize KL divergence with the 
>  intractable "real" model. More specifically, what are the implications of 
>  not including the variational Dirichlet, and just using variational 
>  multinomials over the words (if I'm understanding that correctly)?

Well, the standard variational method for LDA looks at the 
distribution of the document proportions.  This instead looks at
the distribution of word topics.  The standard variational method
has a nice solution, whereas the one of Eq. 4 is an approximation.
But, in our case, for importance sampling of the word topics,
Eq 4 is sampling what we need.  The standard LDA variational approach
doesn't help here, it is sampling the wrong variables.

Not sure I've helped here ;-)

Wray Buntine
Principal Researcher
Statistical Machine Learning

NICTA | Locked Bag 8001 | Canberra ACT 2601
T +61 2 6267 6323 | F +61 2 6267 6230
www.nicta.com.au | wray.buntine at nicta.com.au

>From imagination to impact.

More information about the Topic-models mailing list