I’d like to share my ten key SJT design lessons:
I – Match to job competencies
II – Conduct a range of job analyses
III – Probe for scenarios AND answers
IV – Follow the ‘Rules of Item Writing”
V – Score both SJT answer options
VI – Leverage a subject-matter expert trial stage
VII – Include realism rating scale
VIII – Maximise your trial numbers
IX – Validate
X Update your norms/equal opp’s analysis
I. Match to job competencies
You can buy off-the-shelf SJTs which cater to your industry or which are specific to a function such as sales. However, you then lose that one bit of magic that really makes an SJT special. This is that each question can be crafted to assess critical role behaviours. Having role competencies at your disposal, better yet behavioural indicators is helpful in scenario design. Although…
II. Conduct a range of job analyses
You should run critical incidents role analysis to capture all the critical behaviours: defined as those that lead to successful or unsuccessful workplace outcomes. For example, a happy or an unhappy customer interaction. With, or without, a set of up-to-date competencies for your role, you must schedule as many role analysis sessions at the start of your design process as time (and money!) will allow.
If there are national – or international – role differences then it’s vital, particularly in the US, to document how your job analysis plan captures representative sample(s) for each role being assessed. Logistical pros and cons will certainly dictate the balance of role analysis formats you adopt. I usually recommend a blend of face-to-face strategic interviews with senior stakeholders; a telephone-based approach for job incumbents, but running separate focus groups for their line and regional managers.
III. Probe interviewees (for a range of options)
A few people in your sample will provide most of the information you need. Simply must include the obligatory Pareto Law quote here and say it’s exactly 80% of what you need. Take your time and probe these individuals carefully, going beyond the critical scenarios to explore possible correct and incorrect responses to that situation. Ideally also what a less experienced colleague may do in that situation. Then voila, your interviewee has written the SJT question for you!
IV. Follow the “Rules of Item Writing”
More seriously there’s lots of “rules” to writing clear and concise SJT scenarios and answer options. Too many to cover here in detail. Suffice to say that for each broken item writing rule, your SJT’s validity will suffer that little bit more.
V. Score both answers
When designing the scoring key/format, my preference is usually to score both the Most and the Least answers chosen. This has always worked well in terms of variance and final item selection.
VI. Include an SME Trial
A subject-matter expert (SME) review is vital. It must be done pre-trial so that any unsuitable scenarios are discredited before any valuable trial time is wasted on them. Do schedule in your own editorial time since you need to prepare a finessed Role Incumbent Trial version next.
Aim for 15+ subject matter experts from across the business who have different selection, managing and developing responsibilities for the role being assessed. You can sell this process to senior managers as “the real role experts”, thereby enhancing their engagement in your entire SJT design project!
VII. Check scenario realism
Your trial phase gives you the opportunity to get respondents to rate how realistic the scenario/answer options are. This is invaluable at the trial data analysis phase.
VIII. Maximise trial numbers
Aim for 300 current role incumbents to trial your SJT. Do extend the trial period if necessary to achieve this. However, such a high number is needed to ensure that you have sufficient numbers within each subgroup (for your equal opportunities analysis).
To set accurate cut-off scores, it’s advisable to have collected a form of criterion validity evidence. You are likely to have two samples: your trial group and your validation sample. If the world was a fair place these two would have the same subjects in. In reality, they don’t, and you need to focus on those trial participants for whom you also have job performance data.
X. Updated norm group
My final, key lesson is to use an enhanced norm group as soon as you can, ideally from the start. Once you have sufficient numbers of “live applicants” this is the best norm group to use and any cut-off/equal opportunities analysis can be adjusted accordingly.
Numerical Reasoning book reviews
Brilliant book. Read it from cover to cover. Don’t skip the chapters. I know it can seem patronising, the way he breaks it down, but I’d recommend reading it through.
Very clear layout to follow, questions advance over the chapters and it sets a steady pace. This book helped me immensely to get all the practice I needed within 3 days.