Reliability Is Defined, Within Psychometric Testing

860 Words4 Pages
Reliability is defined, within psychometric testing, as the stability of a research study or measure(s). Reliability can be examined externally, Inter-rater and Test-Retest, as well as internally; which is seen in internal consistency reliability methods. It’s best to establish inter-rater reliability outside of the context of the study’s measure. If the reliability is incorporated in the measure and is extremely low, this can cause serious issues for the researchers going forward. It is highly recommended that a pilot study be done to establish the inter-rater reliability. It is also a good idea to periodically assess the inter-rater reliability if the study will be conducted for an extensive amount of time. There is a difference in how to assess categorical versus continuous measures’ inter-rater reliability. If the measure is a compilation of categories, then the researchers could have the raters mark each category that is applicable to the observations. From those selections, the research team should calculate the percentage of agreement among the raters. However, the research team could also calculate Cohen’s Kappa, for a more “stable” percentage. Cohen’s Kappa Cohen’s Kappa states: where Pr(a) is the observed agreement among raters, and Pr(e) is the conjectural likelihood of chance agreement, using the observed data to calculate the probabilities of the raters hypothetically choosing each category. With this calculation, if the raters are in complete agreement then
Open Document