Please use this identifier to cite or link to this item:
https://doi.org/10.1007/978-3-642-13657-3_42
Title: | Vocabulary filtering for termweighting in archived question search |
Authors: | Ming, Z.-Y. Wang, K. Chua, T.-S. |
Issue Date: | 2010 |
Citation: | Ming, Z.-Y.,Wang, K.,Chua, T.-S. (2010). Vocabulary filtering for termweighting in archived question search. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6118 LNAI (PART 1) : 383-390. ScholarBank@NUS Repository. https://doi.org/10.1007/978-3-642-13657-3_42 |
Abstract: | This paper proposes the notion of vocabulary filtering in a termweighting framework that consists of three filters at the document level, collection level, and vocabulary level. While term frequency and document frequency along with their variations are respectively the dominant term weighting factors at the document level and collection level, vocabulary level factors are seldom considered in current models. In a way, stopword removal can be seen as a vocabulary level filter, but it is not well integrated into the current term-weighting models. In this paper, we propose a vocabulary filtering and multi-level term weighting model by integrating point-wise divergence based measure into the commonly used TF-IDF model. With our proposed model, the specificity of the vocabulary is captured as a new factor in term weighting, and stopwords are naturally handled within the model rather than being removed according to a separately constructed list. Experiments conducted on searching for similar questions in a large community-based question answering archive show that: (a)our proposed term weighting model with multiple levels is consistently better than those with single level for retrieval task; (b)the proposed vocabulary filter well distinguishes salient and trivial terms, and can be utilized to construct stopword lists. © 2010 Springer-Verlag Berlin Heidelberg. |
Source Title: | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
URI: | http://scholarbank.nus.edu.sg/handle/10635/40900 |
ISBN: | 3642136567 |
ISSN: | 03029743 |
DOI: | 10.1007/978-3-642-13657-3_42 |
Appears in Collections: | Staff Publications |
Show full item record
Files in This Item:
There are no files associated with this item.
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.