We published a gamification post over a year ago now, and since it has remained such a hot topic in the world of psychometric assessment – we thought it was worth revisiting, to take a closer look at where this research is at right now.
As an agnostic testing provider, we find ourselves in an interesting position of:
a) Being committed to staying at the forefront of innovation in psychometric testing, but also:
b) Needing to ensure that we carefully review the psychometric properties of each assessment we provide, so that our clients only have access to the most valid and reliable tests on the market.
That being said, let’s review the ‘state of play’ (ha ha, good pun) of gamification in September 2019. As a reminder: gamification can be understood as the application of game-playing elements (such as point scoring or fun activities) to non-game contexts (in this case, assessment).
To prepare this piece, we scoured research databases and also placed a public call-out on LinkedIn, which led to numerous gamification experts sharing their opinions and research papers with us (thank you!).
Support for gamified assessments
Research indicates that gamified assessments can improve user experience; i.e. candidates find them more enjoyable and report positive testing experiences.
This can lead to increased engagement in the recruitment process, and potentially reduce candidate dropout. This is particularly attractive for organisations who view candidates as customers (creating positive associations with their brand).
Some organisations report positive candidate engagement by using gamification in the attraction process: for instance, on their Careers page, to position their brand as progressive and technologically savvy.
Early research supports the potential for traditional cognitive assessments to be ‘gamified’: in the sense of making them more fun or visually appealing (while retaining their predictive validity).
One study found that for a gamified traditional cognitive assessment: candidates reported greater test-taking motivation and reduced test anxiety.
Criticisms of gamified assessments
Thus far, there is limited evidence of predictive validity for gamified assessments (that is, evidence that they predict on-the-job performance), and no published evidence to support their validity in measuring personality.
While some gamified assessments claim to measure personality traits such as decision-making and impulse control, there is no evidence that they predict these things, or that these variables relate to on-the-job performance. These types of ‘cognitive games’ have been used in cognitive psychology for a long time, but they were not designed to measure individual differences.
Many gamified assessments have poor test-retest reliability.
Short gamified assessments don’t measure the candidate very often (provide fewer data points), and in longer gamified tests, the ‘game’ elements can waste time that could be more effectively spent measuring a candidate’s abilities. In other words: sometimes we do not receive as much information as we would from a traditional cognitive assessment. One researcher posed the question: do the gains (improved candidate experience, or marketing value) outweigh the lost information?
It is false for an assessment to claim that it is a “neuroscience game”, unless it actually involves measuring brain structure or activity via a brain scan (e.g. FMRI, CT, EEG).
Assessments that measure reaction time (for instance, by quickly pressing a keyboard button) can be considered ‘cognitive games’ instead. However, check whether the test measures actual cognitive ability (for instance, verbal, numerical, or abstract reasoning), OR cognitive areas like short-term memory and visual attention – which have much weaker predictive validity in a recruitment context. Many gamified assessments also measure numerous cognitive areas at once; research demonstrates that this can make it difficult to isolate performance on separate cognitive domains.
Be wary of poorly designed gamified assessments that are not grounded in psychology theory, and/or were not created by psychometricians or psychologists.
Due to gamification hype over the past 5 or so years, a number of tests have been created that are not really assessments at all: rather, they are games that attempt to capture some assessment data along the way. When the gamification boom began, Gartner predicted that by 2014, 80% of gamification applications would fail to meet business objectives due to poor design. Gamification also has the potential to invalidate an assessment if used improperly. Always ask to see the Technical Manual.
Final food for thought
A thought leader in this space recently described his attitude towards gamification as ‘cautious optimism’: in regard to the scope for traditional cognitive assessments to be gamified, but we need to conduct (and publish) further research, including larger validation studies. It is probably also fair to say that since gamified assessments are relatively new, it may naturally be some time before we can access sufficient validity studies and research papers.
One researcher suggested that there may be other ways to deliver a positive candidate experience, without sacrificing validity. For instance: research indicates that providing personality assessment feedback is an effective way to optimise the candidate experience.
Another consideration: is your gamified assessment accessible to candidates with disability? Some tests can present concerns here: for instance, games requiring differentiation between different colours disadvantage individuals with colour vision deficiency. We came across a very interesting US-based solution that asks candidates upfront whether they have ADHD, dyslexia or colour vision deficiency, and then based on the candidate’s response, automatically adjusts the test accordingly. From a diversity perspective, it is also worth asking the gamified testing provider about availability of different norm groups.
A recent systematic review of gamification in cognitive assessment noted that creating a gamified test is a constant balancing act: making the test engaging while not undermining its scientific validity (including the unknown, potentially deleterious impact of specific game mechanics). One of the thought leaders we spoke with summed up the current research landscape as follows: “This is a really exciting area – when done well, but it also has the capacity to be done badly.”
Some further reading if interested:
> Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafo, M. R. (2016). Gamification of cognitive assessment and cognitive training: A systematic review of applications and efficacy. JMIR Serious Games, 4, 1-14.
> Callan, R. C., Bauer, K. N., & Landers, R. N. (2015). How to avoid the dark side of gamification: Ten business scenarios and their unintended consequences. In Reiners, T., & Wood, L. (eds.), Gamification in Education and Business. Springer: Cham.
> Landers, R. N. (2018). Gamification misunderstood: How badly executed and rhetorical gamification obscures its transformative potential. Journal of Management Theory, 1-4.
> AIHR Live gamification discussion with Ryne Sherman, Chief Science Officer at Hogan Assessment Systems (May 2019): https://www.analyticsinhr.com/blog/aihr-live-episode-6-with-ryne-sherman/