Evaluating N-gram based evaluation metrics for automatic keyphrase extraction

Abstract

This paper describes a feasibility study of n-gram-based evaluation metrics for automatic key phrase extraction. To account for near-misses currently ignored by standard evaluation metrics, we adapt various evaluation metrics developed for machine translation and summarization, and also the R-precision evaluation metric from key phrase evaluation. In evaluation, the R-precision metric is found to achieve the highest correlation with human annotations. We also provide evidence that the degree of semantic similarity varies with the location of the partially-matching component words.