Combining coherence models and machine translation evaluation metrics for summarization evaluation

Abstract

An ideal summarization system should produce summaries that have high content coverage and linguistic quality. Many state-ofthe- art summarization systems focus on content coverage by extracting content-dense sentences from source articles. A current research focus is to process these sentences so that they read fluently as a whole. The current AESOP task encourages research on evaluating summaries on content, readability, and overall responsiveness. In this work, we adapt a machine translation metric to measure content coverage, apply an enhanced discourse coherence model to evaluate summary readability, and combine both in a trained regression model to evaluate overall responsiveness. The results show significantly improved performance over AESOP 2011 submitted metrics. © 2012 Association for Computational Linguistics.