COUN 6360 Week 7 Test Fairness Analysis Example

What Else Could Move His Score? A Test Fairness Analysis Organized by Threats, Not Groups, Naming Reading Comprehension as the Construct and Weighing Language Background and Time Pressure as Sources of Irrelevant Variance for a Composite Adult Entering Technical Training

Student Name

Walden University

COUN 6360: Assessment in Counseling and Education

Instructor Name

Month Day, Year

What Else Could Move His Score? A Test Fairness Analysis Organized by Threats, Not Groups, Naming Reading Comprehension as the Construct and Weighing Language Background and Time Pressure as Sources of Irrelevant Variance for a Composite Adult Entering Technical Training

The Construct

The workforce center wants to know whether Mr. Ortega, the composite 52-year-old client followed in recent weeks, can read the training program's materials, which include equipment manuals, safety procedures and written tests. The construct the chosen test is meant to measure is reading comprehension: the ability to understand and draw inferences from written English text. Fairness, in the sense used in the testing standards' chapter on fairness, concerns whether scores mean the same thing for different test takers, which requires asking what, other than reading comprehension, could raise or lower his score.

The analysis is organized by threats rather than by demographic groups. Membership in a group does not by itself make a test unfair to a person; what matters is whether something irrelevant to the construct affects that person's score.

Threat 1: Language Background

Mr. Ortega grew up speaking Spanish at home and learned most of his English at work. Two aspects of that background could affect a reading comprehension score in ways unrelated to his ability to read training materials.

First, vocabulary. Much of his English vocabulary is work vocabulary: tools, procedures, safety terms. The test's passages, drawn from general academic and literary sources, may rely on vocabulary he had less chance to meet, such as words common in school texts but rare on a warehouse floor. If he misses items because he does not know a literary word, the score reflects vocabulary exposure as well as comprehension, and the training materials, which use work vocabulary, may be easier for him than the score suggests.

Second, cultural knowledge assumed by items. Some passages may assume background knowledge, such as references to particular holidays, school experiences or idioms, that a reader raised in a different culture might not share. Abedi (2002), analyzing achievement test data, found that students' language background was associated with their performance, especially on items with heavy language demands, and that the gap between English learners and native speakers narrowed on items with less language load. Helms (1992) argued that standardized ability testing has paid too little attention to whether tests are culturally equivalent for different groups, a concern that applies to item content as well as to norms.

The manual's evidence on this threat is thin. It reports a bias review by an expert panel and an analysis of item functioning by gender and by race and ethnicity, but no analysis for examinees who learned English as a second language.

Threat 2: Timed Conditions

The reading subtest has a time limit. Mr. Ortega has described a lifelong difficulty reading under time pressure, which he has never had formally evaluated. He says he reads carefully and slowly, and that in his job he could take the time he needed. If the time limit prevents him from finishing, the subtest becomes partly a test of reading speed. Lu and Sireci (2007) describe speededness as a threat to validity when time limits keep substantial numbers of examinees from fully considering all items on a test not intended to measure speed, and recommend that test users examine whether speed is part of the construct.

For training readiness, speed matters only in part. Workers do need to read safety notices promptly, but most training reading, such as manuals and procedures, can be done at the reader's pace. The manual reports that most examinees in the norm sample finished within the time limit, but it does not report completion rates for older adults or second-language readers.

What Can Be Done

Several steps would reduce the threats without abandoning the test. The counselor can record whether Mr. Ortega finishes the reading subtest and, if he does not, report that the score may underestimate his comprehension. A supplementary untimed reading of a passage from the training program's own materials, with comprehension questions written by the training staff, would provide criterion-related evidence on the actual task. A brief screening for a possible reading difficulty, followed by referral for evaluation if indicated, would address the history of slow reading. And the report should state that the norms do not identify second-language learners.

Conclusion

Both threats are plausible and neither is answered by the manual's evidence. Neither makes the test unusable, but both mean the score should be read as a conservative estimate of his reading comprehension for training purposes, interpreted alongside evidence from the training materials themselves.

References

Abedi, J. (2002). Standardized achievement tests and English language learners: Psychometrics issues. Educational Assessment, 8(3), 231-257. https://doi.org/10.1207/S15326977EA0803_02

Helms, J. E. (1992). Why is there no study of cultural equivalence in standardized cognitive ability testing? American Psychologist, 47(9), 1083-1101. https://doi.org/10.1037/0003-066X.47.9.1083

Lu, Y., & Sireci, S. G. (2007). Validity issues in test speededness. Educational Measurement: Issues and Practice, 26(4), 29-37. https://doi.org/10.1111/j.1745-3992.2007.00106.x