Publications -> Conference Papers

Online Multimodal Co-indexing and Retrieval of Weakly Labeled Web Image Collections


Authors: L. Meng, A.-H. Tan, C. Leung, L. Nie, T.-S. Chua, and C. Miao
Title: Online Multimodal Co-indexing and Retrieval of Weakly Labeled Web Image Collections
Abstract: Weak supervisory information of web images, such as captions, tags, and descriptions, make it possible to better understand images at the semantic level. In this paper, we propose a novel online multimodal co-indexing algorithm based on Adaptive Resonance Theory, named OMC-ART, for the automatic co-indexing and retrieval of images using their multimodal information. Compared with existing studies, OMC-ART has several distinct characteristics. First, OMCART is able to perform online learning of sequential data. Second, OMC-ART builds a two-layer indexing structure, in which the first layer co-indexes the images by the key visual and textual features based on the generalized distributions of clusters they belong to; while in the second layer, images are co-indexed by their own feature distributions. Third, OMC-ART enables flexible multimodal search by using either visual features, keywords, or a combination of both. Fourth, OMC-ART employs a ranking algorithm that does not need to go through the whole indexing system when only a limited number of images need to be retrieved. Experiments on two published data sets demonstrate the efficiency and effectiveness of our proposed approach.
Keywords: Hierarchical image co-indexing; Multimodal search; Online learning; Clustering; Weakly supervised learning
Conference Name: Annual ACM International Conference on Multimedia Retrieval (ICMR’15)
Location: Shanghai, China
Publisher: ACM
Year: 2015
Accepted PDF File: Online_Multimodal_Co-indexing_and_Retrieval_of_Weakly_Labeled_Web_Image_Collections_accepted.pdf
Permanent Link: http://dx.doi.org/10.1145/2671188.2749362
Reference: L. Meng, A.-H. Tan, C. Leung, L. Nie, T.-S. Chua, and C. Miao, “Online multimodal co-indexing and retrieval of weakly labeled web image collections,” in Proceedings of the Annual ACM International Conference on Multimedia Retrieval (ICMR’15). ACM, June 2015, pp. 219–226.
bibtex: 
@inproceedings{LILY-c41, 
   author	= {Meng, Lei and Tan, Ah-Hwee and Leung, Cyril and Nie, Liqiang and Chua, Tat-Seng and Miao, Chunyan},
   title	= {Online Multimodal Co-indexing and Retrieval of Weakly Labeled Web Image Collections},  
   booktitle	= {Proceedings of the Annual ACM International Conference on Multimedia Retrieval (ICMR'15)}, 
   year		= {2015}, 
   month	= {June}, 
   pages	= {219-226}, 
   location	= {Shanghai, China},
   publisher	= {ACM},
}