THE DEVELOPMENT OF A READING COMPREHENSION TEST IN ENGLISH FOR TERTIARY-LEVEL STUDENTS IN THAILAND

Abstract

The primary purpose of this study was to develop and validate a reading comprehension test in English for tertiary-level diploma students in Thailand. Two principal concerns were considered in the design of a reading comprehension test: one was that in the choice of test design it should have a beneficial backwash effect and at the same time validly and efficiently assess students' ability to handle real-life language use. The other was that the materials employed in the test should represent the reading tasks that students are likely to perform in real-life communicative contexts. To achieve these objectives, an English reading comprehension measure based on the objectives of a course in a particular college and students' needs was so designed that it was made up of test items assessing various reading skills identified as essential in the foreign language teaching literature. At the pilot stage, a preliminary version of the test was administered in March 1996 to a small sample of 42 students from Rajamangala Institute of Technology Nakhon Si Thammarat in Thailand. After an analysis of the results, a revised version of the test was administered to a larger sample of 210 students in July 1996 in the main study. Two types of information, quantitative and qualitative, were collected from the students. For the quantitative data, several analytical procedures were applied in order to determine the psychometric characteristics of the test with regard to its validity and reliability. A retrospective method was employed to collect qualitative data covering students' experience of responding to the test items and their reactions. Finally, a one-way analysis of variance (ANOVA) and post hoe comparisons (using the Scheffe test) were used to investigate the differences between groups of students from different disciplines in their performance on the reading comprehension test. Analysis of the results of the main study indicated that the overall test had attained a reliability of .75, which was acceptable. In addition, an item analysis revealed that most of the items were appropriate in terms of difficulty level except for a few items, particularly those that were developed to form a new subtest which replaced an ineffective one in the tryout, which needed to be further revised. Findings from item response theory (Rasch analysis) indicated that item statistics fitted the statistical model well. The reading comprehension test results correlated, to some extent, (i.e. r = .64, p < .001), with scores of the mid-term English examination taken at about the same time, which provided some evidence of the concurrent validity of the new instrument. A factor analysis showed that the test seemed to have three main underlying traits - namely global inferencing, vocabulary skill, and lexical inference. The qualitative data gathered from the interviews revealed that students performed well on the matching items and short-answer items but poorly on the gap-filling items and that they needed certain strategies in taking a reading comprehension test. Lastly, the results of the ANOVA and the Scheffe test indicated that students in the disciplines of business computer performed better on the reading comprehension test than those in animal husbandry and machinery. These results bore out the differences in the experiences of these students in learning English over the years (particularly the secondary school and the vocational college groups). The study has demonstrated that the reading comprehension test as constructed is a viable evaluative instrument for tertiary-level diploma students learning English in Thailand.