The Effect of Test Specifications Review on Improving the Quality of a Test

Reviewing the test specifications to improve the quality of language tests may be a routine process in
professional testing systems. However, there is a paucity of research about the effect of specifications
review on improving the quality of small-scale tests. The purpose of the present study was twofold: how
specifications review could help improve the validity of a test in the context of assessment for learning
(AFL) and to what extent qualitative review of items can identify poor ones. To this end, a group of
trained test specifications reviewers (No = 7) provided feedback on the specifications of a test and the
quality of the items. Analysis of feedback showed that pedagogical concerns naturally emerged during
the specifications review and led to concrete suggestions on how the specifications could be revised so
that the resulting test could become more useful in a classroom context. Moreover, the test items were
administered to a group of (No = 40) test-takers and a set of quantitative item analyses was carried out.
Comparison of the results of qualitative judgment of reviewers with the quantitative analyses showed
about 38 % of the items suggested for revision by the reviewers were also identified as poor by the
classical test theory (CTT) analysis. The findings highlight the potential of specifications review as part
of the a priori validation of tests in small-scale assessments where conducting statistical analysis is not
usually feasible.