THE COGNITIVE LEVELS OF READING COMPREHENSION TEST ITEMS : AN EXPLORATORY STUDY

Abstract

The study sought (a) to investigate the perceptions of a sample of English Language teachers with regard to the current practice of distinguishing between lower-order and higher-order skills in developing reading comprehension test questions, and (b) to find out the extent of agreement among teachers and the extent of relationship, if any, between the teachers' perceptions and item difficulty as demonstrated by students' performance on those test items defined as assessing lower-order or higher-order skills. This was an exploratory study, with a sample of test performance obtained from 150 secondary three students of a wide range of ability in the English Language. The study was prompted by recent research which questioned the accuracy and reliability of teachers' perceptions in the construction of reading comprehension test questions. Based on the assumption that reading skills can be identified, taught and tested in second language reading, test items are often classified into those assessing higher and lower-order skills. Recently, however, doubt has been cast on these assumptions, in that it has proved difficult to get experienced teachers to agree on the skills tested by individual items and the relative difficulty levels of those skills. The design of the study comprised two phases - the judgemental and the empirical. In the judgemental phase, a sample of teachers was asked to match each of the identified skills to individual test items and to classify the same items into those that assessed lower or higher-order skills. In the empirical phase, a 20-item single reading comprehension test was administered to 150 subjects who were divided into two groups based on their first semestral assessment results in reading comprehension. In addition, a pilot study preceded the main study in order to gauge the effectiveness of the research instruments. Data obtained from the judgemental and the empirical phases provided the basis for computing classical reliability estimates, difficulty and discrimination indices, and item response theory (IRT) estimates of item parameters. The data obtained from the empirical phase were then examined in the context of perceptions of teachers. The findings derived from the analysis of the data collected revealed that 1) the teachers were able to achieve a high consistency of agreement among themselves regarding the relative difficulty levels of the skills tested by the test items; and 2) there was a significant relationship between teachers' perceptions and item statistics. The study demonstrated that there was empirical support for teachers' perceptions with regard to the practice of making a distinction between lower-order and higher-order skills in developing test questions in reading comprehension.