Publications -> Conference Papers

A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution


Authors: S. Li, J. Zhu, and C. Miao
Title: A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution
Abstract: Most existing word embedding methods can be categorized into Neural Embedding Models and Matrix Factorization (MF)-based methods. However some models are opaque to probabilistic interpretation, and MF-based methods, typically solved using Singular Value Decomposition (SVD), may incur loss of corpus information. In addition, it is desirable to incorporate global latent factors, such as topics, sentiments or writing styles, into the word embedding model. Since generative models provide a principled way to incorporate latent factors, we propose a generative word embedding model, which is easy to interpret, and can serve as a basis of more sophisticated latent factor models. The model inference reduces to a low rank weighted positive semidefinite approximation problem. Its optimization is approached by eigendecomposition on a submatrix, followed by online blockwise regression, which is scalable and avoids the information loss in SVD. In experiments on 7 common benchmark datasets, our vectors are competitive to word2vec, and better than other MF-based methods.
Keywords: 
Conference Name: Conference on Empirical Methods in Natural Language Processing (EMNLP'15)
Location: Lisboa, Portugal
Publisher: SIGDAT
Year: 2015
Accepted PDF File: A_Generative_Word_Embedding_Model_and_its_Low_Rank_Positive_Semidefinite_Solution_accepted.pdf
Permanent Link: http://www.emnlp2015.org/proceedings/EMNLP/pdf/EMNLP183.pdf
Reference: S. Li, J. Zhu, and C. Miao, “A generative word embedding model and its low rank positive semidefinite solution,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’15). SIGDAT, September 2015, pp. 1599–1609.
bibtex: 
@inproceedings{LILY-c49, 
   author	= {Li, Shaohua and Zhu, Jun and Miao, Chunyan},
   title	= {A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution},  
   booktitle	= {Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'15)}, 
   year		= {2015}, 
   month	= {September}, 
   pages	= {1599-1609}, 
   location	= {Lisboa, Portugal},
   publisher	= {SIGDAT},
}