Differences in Rater Behavior

1070 WordsJul 8, 20185 Pages
Differences in rater behaviors are among the factors responsible for variability in the decision making process(DMP) during ratings. The interference of either the rater rating style or rater experience determines the validity and reliability of the rating score and the rater themselves. Factors related to rater inconsistencies identification and measurement in DMP is necessary to avoid factors underlying variability in decision making process . Several studies have identified rater proficiency level, rater experiences and tasks as its factors. The purpose of this paper is to critically review two articles that contribute to describe insights of rater behavior related to the factors studied. Barkaoui (2010) ‘Variability in ESL Essay…show more content…
Unlike Baker, Barkaoui limits the think aloud protocol as its only quantification of the rating scores, underestimating interference of the think aloud protocol in the rating process(Barkaoui,2010,57). The difference in the number of participants may restrict the author from applying other counterbalance methods in this case. Another important issue to point out is the data collection and presentation of the studies. Barkaoui (2010) rater samples includes natives and non-natives, qualities that may distort the study results as an external factor(p.57). It is in doubt that such differences has no impact on rating scores collected since other studies have claimed them as a factor that has an impact on raters rating performance. Regarding the data presentation, Baker (2010) studies further demonstrates more distinctive details and explicit insights in each raters behavior details. It successfully illustrate the data collected during the write aloud protocols in the findings discussions (Baker,2012,p.237-238). Barkaoui (2012) experimental study disregarded the importance of participants comments by excluding these data. The think aloud elicited data were presented in the appendix with limited reference in
Open Document