[Topic-models] Intuition behind CTM and DTM
Erik Sudderth
sudderth at eecs.berkeley.edu
Mon Nov 24 14:56:03 EST 2008
Thanks for the very clear explanation of the logistic normal
distribution, David. I noticed a small bug in your R code: you need to
multiply by a square root of the covariance matrix, not the matrix
itself, to sample from a Gaussian. The following code does this:
## zero-mean logistic normal
rlogisticnorm <- function(covariance = matrix(c(2, 0.5, -0.5, 0.5, 2,
0.5, -0.5, 0.5, 2), nrow=3)) {
n <- dim(covariance)[1]
covarianceChol <- chol(covariance)
result <- exp(t(covarianceChol) %*% rnorm(n))
result / sum(result)
}
Best,
Erik
David Mimno wrote:
> On Mon, Nov 24, 2008 at 12:14:11AM -0700, Lei Tang wrote:
>> 1. In correlated topic models, the topic proportion is sampled from a
>> logistic normal distribution instead of Dirichlet as in LDA. I didn't quite
>> understand the intuition behind such a modeling. Why is logistic normal
>> distribution has such power?
>
> There are two primary advantages:
>
> First, covariance. (See John Aitchison's work for a more detailed
> discussion.) Let's say I have a corpus with three topics: sports (team,
> player, league), politics (weapons, trade, president), and negotiation
> (meeting, deadline, agreement). Both sports and politics occur with
> negotiation, but sports and politics rarely cooccur.
>
> With a Dirichlet, all I can say is how often I expect each topic to occur
> (the values of the parameters in proportion to each other) and how much I
> expect any given document to follow those proportions (the sum of the
> parameters, where larger = less variance). With a logistic normal, I can
> set up a covariance matrix with positive covariance between sports and
> negotiation but negative covariance between sports and politics.
>
> Second, there are very well studied models for time-series and
> spatio-temporal data in continuous spaces. These usually aren't applicable
> to count data like words, but if you can represent the word counts as
> derived from a real-valued hidden variable, Kalman filtering and dynamic
> linear models become available.
>
> Here are two R functions that might help give some intuition for the
> parameterization and the behavior of Dirichlets and logistic normals:
>
> ## Dirichlet
> rdirichlet <- function(alpha = c(1.0, 1.0, 1.0)) {
> n <- length(alpha)
> result <- rep(0, )
> for (i in 1:n) {
> result[i] <- rgamma(1, alpha[i])
> }
>
> result / sum(result)
> }
>
> ## zero-mean logistic normal
> rlogisticnorm <- function(covariance = matrix(c(2, 0.5, -0.5, 0.5, 2, 0.5,
> -0.5, 0.5, 2), nrow=3)) {
> n <- dim(covariance)[1]
> result <- exp(covariance %*% rnorm(n))
>
> result / sum(result)
> }
>
> -David
> _______________________________________________
> Topic-models mailing list
> Topic-models at lists.cs.princeton.edu
> https://lists.cs.princeton.edu/mailman/listinfo/topic-models
More information about the Topic-models
mailing list