Make Personality Tests More Relevant – Put Them In Context
Personality tests are valid – to a point. It’s almost undisputed that the major personality tests generally tell us something useful about a person. But how much they tell us is usually pretty small – they make us only slightly smarter about the person taking the test.
So, how do you change the tests so they make us more than just slightly smarter? Context, context, context! One name used for context is “frame of reference.” So, instead of a context-free item like “I pay attention to details,” you go with an item like “I pay attention to details at work.”
Lievens and his colleagues challenged the conventional wisdom about why context works. It has been widely argued that contexts reduces the variability among test-takers, making it less likely that some of the test takers will answer a question across all contexts and others will self-select a context in which they interpret the test questions.
Lievens et al suggest – then test – an alternative explanation. They do not deny that context reduces between-person variability, but they suggest that the impact is minimal. They point to within-person variability as the key threat to validity that frame-of-reference reduces. The thinking behind this explanation is that, when responding to an item on a test, a person will conduct a selective search of his/her memory to help them respond to the item, and that providing the context for that memory search helps assure that the person will be more consistent with himself/herself in the responses. Thus reliability, and therefore validity, of the measure increases, not because different people are more likely to answer questions consistently with one another, but because each person is more likely to answer them consistently with themselves.
In a pair of studies, the researchers found the alternative explanation – focusing on within-person variability – to provide a better explanation of how context helps improve tests. Besides giving us a clearer theoretical understanding of how people think when they take tests, this research will help designers of test make sure the tests are not only reliable, but also valid, by making sure they select the correct frame of reference. The researchers found that using an incorrect frame of reference – one that is conceptually irrelevant to the criterion -- actually decreases validity.
From: Lievens, F, De Corte, W., & Schollaert, E. (2008). A closer look at the frame-of-reference effect in personality scale scores and validity. Journal of Applied Psychology, 93(2), 268-279.