Improving the validity of forced-choice assessments through questionnaire optimization

Resumo

Whereas the assessment of non-cognitive traits with single-statement items is knowingly susceptible to response biases such as acquiescence, social desirability responding and faking, the forced-choice (FC) format has been consistently found to reduce such effects. Nonetheless, the FC format has long been considered problematic for providing ipsative scores. This dissertation investigates ways to improve the validity of non-cognitive assessments, trying to reduce the susceptibility to response biases by using the forced-choice format, while allowing to provide with normative comparisons between respondents. Specifically, three studies are presented. Using real data, the first study compares the convergent and criterion-related validity of personality scores for the same set of items presented in two formats: 1) as single-statement items and 2) in forced-choice pairs. Additionally, single-statement items are modelled accounting for social desirability responding and acquiescence using a compensatory IRT model, whereas the forced-choice responses are modelled using a dominance forced-choice model. The second study presents a genetic algorithm (GA) to optimize the assembly of forced-choice questionnaires aiming to maximize the reliability of the FC scores. A simulation study is conducted, comparing the questionnaires formed using this algorithm and the average characteristics of questionnaires formed at random as an indicator of the expected results when not optimizing block assembly. An online application was made available for practitioners to facilitate the construction of optimally assembled forced-choice questionnaires. Finally, the third study focuses on maximizing the reliability and validity of the scores in multidimensional adaptive assessments. Specifically, a simulation study addresses the effect of 1) different bank assembly (with a randomly assembled bank, an optimally assembled bank with the GA, and blocks assembled on-the-fly), and 2) block selection rules over the estimate accuracy, ipsativity and overlap rates