Web Information Paper Review (3)

Mark Steyvers, Tom Griffiths:Probabilistic Topic Models, Latent Sematic Analysis: A Road to Meaning

Links: gibbs抽样方法学习笔记, 主题模型-LDA浅析, LDA-math-MCMC 和 Gibbs Sampling

This paper mainly introduces the Probabilistic Topic Models, which means understand the documents are expressed in terms of the probabilistic topics and the topics are probabilistic distribution of words. In comparison with the traditional semantic analysis method like TF-IDF, this method consider the latent semantic relationship beyond the words. This model also called as Latent Dirichlet Allocation. It share a lot of commons with the LSA.

Probabilistic Topic Models can help us think the document as a joint distribution of several topics and topics are distributions over a cluster of words with correlated relationship. So the generation of the document can be viewed as: choose several topics then concrete the words in each topics together to form a document. In probabilistic view, it is expressed as: P(word=i | document) = ∑ p(word=i | topic=j) * p(topic = j | document). We can get the P(word=i | document) from the exist accumulated document and words’ data. The goal of Probabilistic Topic Models is to learn from these data and obtain the p(word=i | topic=j) and p(topic = j | document) as the model parameters.

The generative model of this is called as Latent Dirichlet Allocation. The Dirichlet distribution is the probabilistic description of the multinomial distribution. We can understand it as it descript the possibility of a certain combination of topics with each has the show up possibility in a certain model. So it is a probabilistic model to describe another probabilistic model. As the example in the paper, if we have 3 topics, we can use a simplex to show our topic combination possibility distribution. The plate graphic model also give us clear idea of the topic model. The alpha and beta are hyperparameters which are the prior observation of the Dirichlet distribution. Alpha is the topics’ distribution parameter over document and the beta is the word’s distribution parameter over topic. These two are obtained from the prior collected data, so that they are stay constant. Theta is the P(theta | alpha) and Phi is the P(phi | beta). The D plate means the sampling process in the topics until we get the document and the the T plate means the sampling process in the words until we get the topic T. By using this way we can put the intermediate topic model into the same simplex of the word and document distribution simplex. So this can help us better understand the meaning of the topic in documents. Also this LDA has relationship with the LSA which also study the latent semantic relationship under the words. LSA is study the relationship in the word perspective while LDA is study the relationship under the word’s distribution - topic.

Our goal is to obtain the theta and the beta under the prior collected data. The method to achieve this is by using the Gibbs Sampling method. It is a method of Markov Chain Monte Carlo. MCMC means that in a Markov Chain, which a sequence of events will happen and the events are depends on prior events and they are not independent, after n times of iteration, the probabilistic model to describe each event will converge to a certain probabilistic model. Like in the LDA case, the deserved alpha and the beta is the converged probabilistic model we want to get. The Gibbs sampling is first use the prior collected data’s distribution to generate an artificial document as initial data. And then use the axis switch method to update each distribution parameters. After one update of the whole params, we get an event of Markov Chain. We can keep repeat this update process until the params are converge to a certain value. MC must be converge to some specific values, so this value is the our approximation of the original topic and word distribution’ models.

From this paper, I learn a new way to study the document. We can view it as the collection of some topics and by study the exist data, we can reconstruct the model of the topics and the words. This can give us a better view of the semantic analysis.