A testing organization administers an exam that contains new and reused items. As part of a data forensics analysis, each candidate’s performance on reused items versus new items is compared to detect potential preknowledge of the reused items (New-Old Difference Statistic).

Recently, a training program had a high detection rate by the New-Old Difference Statistic, and several candidates in that program had highly anomalous score differences between the reused items and the new items. Historical data showed the anomalies at this program steadily increased over time. This presented compelling evidence that items were being harvested and disclosed to candidates in this program.

An investigation traced the item harvesting and disclosure to a review course. Also, it was discovered that this review course was operating in multiple training programs. In response, the testing organization retired all previously-used items, which required a substantial amount of item development.

The first part of this session discusses the data forensics and investigative findings that led to the decision to retire all previously-used items. The second part discusses the organization’s efforts to quickly develop new items, an evaluation of the new exam’s performance, and policies implemented for refreshing test content in the future.