International Conference on Assessment and Learning (ICAL)

Publication Date

10-2022

Download

Download Full Text (196 KB)

Keywords

Test validity, Test content, Test construction, Literature review, Measurement

Subjects

Content validity, Test validity, Test content, Test construction, Literature review, Measurement

Disciplines

Educational Assessment, Evaluation, and Research

Comments

Paper presented at ICAL 2022 — Transforming Assessment and Learning: Making the System Work!

13 - 15 October 2022, Bali, Indonesia

Description

Validity is a crucial issue in test development because it represents the accuracy of test score interpretation in describing the measured attribute or construct. Among validity evidence that can be collected to support test score interpretation, the one based on test content incited controversies. The content-based validity evidence includes analyses of the relationship between test content and the construct the test purported to measure. Some authors opposed the use of such evidence in the validation process, arguing that the evidence can only provide hypotheses to be tested in the subsequent processes. Other authors argued that the content validation process provided meaningful information regarding test validity, particularly tests based on content such as achievement tests. Unfortunately, literature regarding such issues hitherto has not addressed the controversies adequately. Furthermore, several techniques for quantifying the content-based validity evidence have been proposed without any other article reviewing and evaluating them in comparison with the others. The current study was conducted to examine propositions made by prominent authors regarding validity based on test content and to compare proposed quantification techniques. Using Bielefield Academic Search Engine, 1,841 articles with “content validity” in their titles, however only 28 of them met inclusion criteria and thus reviewed. The reviews showed that there were three positions regarding content-based validity evidence: (1) content validity is sufficient as sole evidence supporting test scores interpretation, (2) evidence can be based on the relationship between content and construct but support from other types of evidence is needed, and (3) evaluating representativeness of construct by test content is an important process, but it cannot satisfactorily provide validity evidence. As for the quantification techniques, there were four techniques proposed by different authors. Each focused on a different aspect of the relationship between construct and test content, such as whether or not test content is relevant (e.g., Aiken’s V and Polit’s CVI), essential (Lawshe’s CVR), or discriminant (Dixon and Johnston’s Discriminant Content Validity) to another construct. All but one of the techniques were variations of measures of agreement between experts regarding how well an item represented a construct. Only one technique used statistics comparing means of judgments on the content relevance to different constructs. Most techniques have been evaluated and revised or corrected by other authors, while Dixon and Johnston’s DCV has not because it was just published currently. The impact of the findings was discussed.

City

Jakarta, Indonesia

Publisher

ACER Indonesia

ISBN

978-1-74286-697-0

DOI

https://doi.org/10.37517/978-1-74286-697-0-01

Validity evidence based on content: Controversies and quantification

Share

 
COinS