[Topic-models] Intuition behind CTM and DTM

Erik Sudderth sudderth at eecs.berkeley.edu
Mon Nov 24 14:56:03 EST 2008


Thanks for the very clear explanation of the logistic normal 
distribution, David.  I noticed a small bug in your R code: you need to 
multiply by a square root of the covariance matrix, not the matrix 
itself, to sample from a Gaussian.  The following code does this:

## zero-mean logistic normal
rlogisticnorm <- function(covariance = matrix(c(2, 0.5, -0.5, 0.5, 2, 
0.5, -0.5, 0.5, 2), nrow=3)) {
	n <- dim(covariance)[1]
         covarianceChol <- chol(covariance)
	result <- exp(t(covarianceChol) %*% rnorm(n))

	result / sum(result)
}

Best,
Erik

David Mimno wrote:
> On Mon, Nov 24, 2008 at 12:14:11AM -0700, Lei Tang wrote:
>> 1. In correlated topic models, the topic proportion is sampled from a
>> logistic normal distribution instead of Dirichlet as in LDA. I didn't quite
>> understand the intuition behind such a modeling. Why is logistic normal
>> distribution has such power?
> 
> There are two primary advantages:
> 
> First, covariance. (See John Aitchison's work for a more detailed 
> discussion.) Let's say I have a corpus with three topics: sports (team, 
> player, league), politics (weapons, trade, president), and negotiation 
> (meeting, deadline, agreement). Both sports and politics occur with 
> negotiation, but sports and politics rarely cooccur.
> 
> With a Dirichlet, all I can say is how often I expect each topic to occur 
> (the values of the parameters in proportion to each other) and how much I 
> expect any given document to follow those proportions (the sum of the 
> parameters, where larger = less variance). With a logistic normal, I can 
> set up a covariance matrix with positive covariance between sports and 
> negotiation but negative covariance between sports and politics.
> 
> Second, there are very well studied models for time-series and 
> spatio-temporal data in continuous spaces. These usually aren't applicable 
> to count data like words, but if you can represent the word counts as 
> derived from a real-valued hidden variable, Kalman filtering and dynamic 
> linear models become available.
> 
> Here are two R functions that might help give some intuition for the 
> parameterization and the behavior of Dirichlets and logistic normals:
> 
> ## Dirichlet
> rdirichlet <- function(alpha = c(1.0, 1.0, 1.0)) {
> 	n <- length(alpha)
> 	result <- rep(0, )
> 	for (i in 1:n) {
> 		result[i] <- rgamma(1, alpha[i])
> 	}
> 	
> 	result / sum(result)
> }
> 
> ## zero-mean logistic normal
> rlogisticnorm <- function(covariance = matrix(c(2, 0.5, -0.5, 0.5, 2, 0.5, 
> -0.5, 0.5, 2), nrow=3)) {
> 	n <- dim(covariance)[1]
> 	result <- exp(covariance %*% rnorm(n))
> 
> 	result / sum(result)
> }
> 
> -David
> _______________________________________________
> Topic-models mailing list
> Topic-models at lists.cs.princeton.edu
> https://lists.cs.princeton.edu/mailman/listinfo/topic-models


More information about the Topic-models mailing list