Situational judgement test design is one of our key psychometric test design specialities.

Our client bespoke Situational judgement test design projects

1. CAPP’s BESPOKE SITUATIONAL ASSESSMENTS 

• Healthcare enterprise client in FTSE 100
• Enterprise banking and insurance clients

2. BUPA situational judgment test

3. Army Officer and Army Soldier SJT’s and role profilers

4. HARAPPA’s situational judgement test designs

• Gen Z-focused educational assessments of work and life skills.
• School version of each competency created too.

5. KENEXA IBM’s situational judgement test design

• Designing personality, situational and aptitude tests for enterprise cients (Boots, Nat West, RBS, Jaeger, Ladbrokes, Whitbread)
• Reviewing personality profile report based on sentient analysis (via Watson).

6. TALENTQ situational judgement test design

• Multiple enterprise finance and retail clients.

7. WE ARE AMBERJACK situational judgement test design

Our other psychometric test design specialities

• Firstly, here are some examples of our specialist situational judgment test designs.
• Secondly, our values situational and competency values-based assessment designs.
• We also specialise in aptitude test design projects.
• Then next our customised leadership assessment projects and pre-employment sift designs.
• Plus, our emotional intelligence assessment designs. and high potential test designs.
• And then next, our top bespoke personality test design projects and our school aptitude test designs.
• Finally, some examples of the bespoke psychometric tests designed recently for clients.

Our situational judgement test design

Rob Williams Assessment Ltd have considerable experience in situational judgement test designs. Our approach to situational judgement test designs is to always design bespoke situational judgement tests based upon a highly detailed role analysis.

Introduction to situational judgement tests

A situational judgment test presents realistic scenarios similar to those that would be encountered when doing a particular job. By using real-life scenarios, the idea is that employers will get a better understanding of how you might operate in the workplace. Typically, candidates must identify the best and the worst course of action from four multiple-choice options. In other words, you must make two “judgments” about what to do and what not to do next in the “situation” presented.

Example situational judgement test

You have two very important deadlines to meet by the end of your working day. However it is becoming clear in your final two hours of working that you are in danger of missing both deadlines?

Select your most preferred and least preferred responses:

(a) Work out what to do. Then prioritise the critical tasks for the time remaining.

(b) Focus on still doing a quality job even if you must miss a deadline.

(c) Speed up your remaining tasks so that you will still be able to meet both deadlines.

(d) Aim to achieve one deadline and to renegotiate the delivery date for the other.

Answer to situational judgement test example question

In this instance, the best response is (a) Whereas, the worst response is (b) Focus on still doing a quality job even if you must miss a deadline. The best outcome is to meet the deadline after making a considered judgement call. Missing the important deadline is the worst outcome.

Best practice SJT design

The SJTs we design follow these best practices in SJT design.

– Firstly, review existing role descriptions and competency framework literature.
– Secondly, conduct job analysis sessions (face-to-face or telephone-based) with a selection of subject matter experts.
– Thirdly, reate draft scenarios encompassing positive and negative behavioural responses
– And then write realistic responses to each draft scenario.
– Also, run a small trial of the draft SJT scenarios with a set of subject matter experts. This is primarily to agree the scoring key.
– And then analyse descriptive statistics for the first trial.
– Run a full trial of the draft SJT scenarios.
– Apply and check the scoring key against the trial SJT test results.
– Analyse descriptive statistics for this trial (mean, SD etc).

Ongoing situational test trends

Branching situational judgement tests 

As psychometric tests have become more commonplace, the bigger users have commissioned their own bespoke situational judgement tests. Rob Williams Assessment has worked on several such projects for High Street banks and for the European Union. Another recent innovation of test developers has been online adaptive tests. With these tests, if you are doing well, you will find that the questions get progressively harder. That can feel like a challenge since you are pushed until you reach the most challenging level you can. This is the level at which you – just like other candidates with your level of verbal reasoning – start to get questions wrong.

The innovative design of shorter and more efficient tests was driven by an increasingly aware of the immediacy of the Internet and our increasing use of emails and social media in short, sharp bursts.  This discourages test takers from spending 30-40 minutes online doing the same questionnaire. Its better for everyone to keep test takers engaged when being tested – not bored!

So what will adaptive tests mean for you as a prospective test taker?  The biggest difference is the shortness of the test. The second major difference is that you will find an adaptive test more challenging. Without getting into their highly technical make-up, the test adapts to your ability level. More specifically it adapts to find the most challenging question that you can answer correctly.

In the past you may have found questions on a test fluctuating in difficulty or generally becoming more and more difficult the further on you get in the test. Consider a test of twenty questions with the first the easiest and the twentieth the most difficult.

Knowledge-based situational judgement tests 

Some or all of the scenarios presented in an SJT can test specific job knowledge. For example, a retail marketing SJT may ask questions about the 3Ps (price, position, promotion) of product marketing. Alternatively both an SJT measuring generic decision-making skills may be used alongside a knowledge-based test.

Video based Situational judgment assessment

Simulated situational judgement tests are increasingly common as recruitment sifts. Adding 2D or 3D workplace scenario graphics brings the situational judgment test scenarios to life. This can only promote the company brand and make employers using simulated situational judgment tests more desirable employers.

UK and US psychometric test publishers have produced both video-based and animated SJT scenarios. Animated SJTs are easier – and therefore cheaper – for global companies to develop.

Bespoke situational test designs

Situational judgement tests (SJTs) have also become prevalent in graduate recruitment. These tests presents scenarios to applicants and asks them to select the best and the worst thing to do next. SJT’s are very popular in the United States due to their excellent record of fairness across different ethnic groups.

Bespoke situational judgement test design benefits

• Strong research evidence – a highly valid and fair test of a range of professional skills.
• High face validity and job relevance

• Reduced adverse impact  – especially compared to aptitude tests such as verbal reasoning tests with their high language loading.
• Cost effective development time – a relatively cost effective testing solution.

Digital situational judgement test formats

Rob Williams Assessment Ltd have worked on many situational judgement tests including the different formats described above: ranking 4-6 response options, choosing the Best and/or the Worst response option, and producing video-based online situational judgement test simulations. Rob Williams Assessment Ltd have experience in developing situational judgement tests in the retail, graduate, healthcare and armed service sectors.

In particular we offer considerable experience in the most effective ways to write SJT scenarios and response options that work, as well as the most useful statistical analyses to conduct of your trial SJT data. This is crucial for ensuring the validity of the SJTs you are developing.

Situational Judgement Test Advantages

SJTs advantages, in any suitable recruitment context, ae varies and numerous. The main ones that make SJTs particularly attractive, for example to sift out graduate applicants, are as follows:

• Easy to administer to large audience
• Relatively strong validity
• Can assess sample of the relevant KSA’s
• Refreshed items possible
• More difficult to fake responses

SJT Design Projects

• Lazard Brothers and AECOM graduate engineers. Situational judgement test validations
• Citibank. Designing blended assessments for retail and call centre operations in American and Mexico. Managing item writing team for SJTs, ability tests, biodata questions etc.
• Citibank telephony customer service agents (Mexico, Malaysia, India) and retail banking roles (North America): manager, assistant manager, bank teller, personal banker.
• European Union. Development of project management and IT skills-based aptitude tests for the European Union.
• British Airways cabin crew, British Airways customer service and call centre operatives.
• BUPA Care Assistants 
• Lazard Brothers SJT validation  and cut-off setting
• AECOM graduate engineers SJT validation and cut-off setting

Ask ROB (expert test developer) your questions by emailing rrussellwilliams@hotmail.co.uk

An Associate Fellow of the British Psychological Society, Rob Williams is a Chartered Psychologist with over 25 years of experience working and designing tests. He is also the author of five psychometric test design books and has worked for the leading global psychometric test publishers including SHL, Kenexa IBM, MBTI, CAPP and SOVA Assessment.

Beneficial SJT Design Project Outcomes

• Best practice was followed throughout the design process in SJT design.
• An SJT was produced which successfully incorporated a range of care home-specific scenarios.
• The most suitable set of scenarios could be hand-picked at the SME panel meeting, as well as gaining buy-in and discussing implementation.

• Providing some scenarios for telephone interview sifts.
• Setting a suitable cut-off and validating the tool.
•  

Situational judgement test applications

Increasingly online situational judgment tests are being used as a recruitment sift in place of a more traditional verbal reasoning test and/or personality questionnaire. Here are some of the advantages that situational judgement tests offer compared to more traditional psychometric tests:

– More efficient to use one psychometric combining elements of both reasoning and personality assessments

– Once a bespoke situational judgment test is developed, it is more cost effective

– Situational judgment tests excel at sifting out high volumes of poorer quality candidates whilst selecting in the high calibre candidates who have scored highly across the range of role competencies

– Fairer assessment in terms of diversity issues since there is minimal adverse impact due to gender or ethnic origin

– Situational judgment tests are particularly good at assessing cultural fit. A well designed situational judgment test can therefore assess organisational values – as well as an organisation’s competencies per role.

– Situational judgment tests (SJTs) can act as a realistic job preview since SJTs and RJPs (RJPs)  have the same test format. Candidates can sift themselves out of a recruitment process at an early stage – often even before applying if the realistic job preview sits freely available on the company website.

– You can purchase generic situational judgment tests from most UK psychometric test publishers. A bespoke SJT is a more effective sift even though it will be more expensive initially.

Situational judgement test design

Why use Situational Judgement Tests?

Candidates prefer SJTs compared to cognitive ability and personality tests.

Traditionally, selection into medical education professions has focused primarily upon academic ability alone.

There are mounting calls to widen access to medicine, to ensure that selection methods do not unfairly disadvantage individuals from specific groups (e.g. regarding ethnicity or socio-economic status) so that the future workforce adequately represents society as a whole.

What are Situational Judgement Tests?

Firstly, situational judgement tests (SJTs) assess individuals’ reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role.

Secondly, in our opinion, SJTs measure pro-social Implicit Trait Policies (ITPs) – shaped by socialisation processes. For example, helping others in need), or disagreeable actions (e.g. advancing one’s own interest at others, expense).

SJTs have significant added value (incremental validity) over and above other selection methods such as knowledge tests, measures of cognitive ability, personality tests and application forms. SJTs have lower adverse impact compared to other selection methods, such as cognitive ability tests.

Are situational judgment tests valid?

Specifically, evidence consistently shows that SJTs used in medical selection has good reliability, and predict performance across a range of medical professions, including performance in general practice, in early years (foundation training as a junior doctor) and for medical school admissions.

SJTs have the benefit of being appropriate both for use in a selection where candidates are novices (i.e. have no prior role experience or knowledge such as in medical school admissions) as well as settings where candidates have substantial job knowledge and specific experience (as in postgraduate recruitment for more senior roles). An SJT specification (e.g. scenario content, response instructions and format) may differ depending on the level of job knowledge required.

Situational judgement test design

Background to situational judgement tests

Situational judgement test questions can be designed around a specific role. Because SJTs are bespoke, candidates rate them highly as a valid application stage.

In fact, there is a type of SJT called a realistic job preview which is used solely to give job applicants a realistic preview of a role.

In our opinion, situational judgment test presents realistic job scenarios. The idea is that employers will get a better understanding of how you might operate in the work place.

• These measure those skills of judgment and evaluation which are vital to any managerial role. Role-specific competencies will be measured, typically customer service, communication and teamwork.
• SJTs are a highly effective means of measuring an applicant’s competency behaviours – i.e., the characteristics that determine how they behave in certain areas.
• SJTs are a fair test of how the candidate would solve a job’s daily challenges.

Situational judgement test design

Situational judgment test design

Different uses of Situational Judgement Tests

• In comparison to simulations and role-playing exercises, SITUATIONAL JUDGEMENT TESTs are often more objective and more cost-effective, so they can provide valid and reliable — and thus trustworthy — ways of measuring judgment in the workplace. SITUATIONAL JUDGEMENT TESTs also allow assessments to operate at higher levels of the Blooms Taxonomy.

• The key to authoring an effective SITUATIONAL JUDGEMENT TEST is to identify a dilemma that requires a choice to be made and where using judgment to resolve that choice is clearly linked to a needed domain of knowledge, skill or competency in the job role. Because an SITUATIONAL JUDGEMENT TEST is only useful if it requires professional judgment to answer well, you must involve Subject Matter Experts in authoring and reviewing SITUATIONAL JUDGEMENT TESTs.

• There are many options and formats for SITUATIONAL JUDGEMENT TESTs. One of the most effective is the Select Most and Least approach, where 4 or more options are presented and the participant must identify the most and least effective or appropriate. This has the advantage of being relatively simple for the participant to consider, while making it but much harder to guess the right answer than it would be to select a single most appropriate response.

• Within regulatory compliance, SITUATIONAL JUDGEMENT TESTs can be used to assess competence and survey how participants think personnel will respond.
• For certification programs, SITUATIONAL JUDGEMENT TESTs provide a relatively inexpensive way of extending coverage to higher-order and softer skills required by participants.
• For pre-hire assessments, SITUATIONAL JUDGEMENT TESTs provide a way for candidates and organizations to get insight on a candidate’s capability and fit for the role.
• And for post-hire training and workforce development, SITUATIONAL JUDGEMENT TESTs give the ability to measure important workforce competencies not easy to assess in other ways.

Designing your SJT scales, scenarios and answer options

Needs may be for a standalone SITUATIONAL JUDGEMENT TEST in the form of several SITUATIONAL JUDGEMENT TEST items or for a blend of SITUATIONAL JUDGEMENT TEST and knowledge-based items (or other content).

Random selecting and ordering of questions can be helpful.   It can be useful to Choose question order randomly to avoid ordering effects and as an anti- cheating measureSelect questions randomly from an item bank to reduce question exposure and as an anti-cheating measure Select different questions on subsequent attempts from those a previous participant saw previously

3. Allow a survey to be presented anonymously.   When presenting SITUATIONAL JUDGEMENT TESTs in a survey, for example in regulatory compliance where you are seeking to assess opinion or likely behavior, it can be useful to make results anonymous. This encourages participants to answer more freely, particularly on sensitive matters.

4. Make it easy to manage the translation of SITUATIONAL JUDGEMENT TESTs.   Because SITUATIONAL JUDGEMENT TESTs involve complex thinking, it is important to present them in a language the participant is very familiar with — so translation is often required. Ensure that you review and pilot all translations to avoid loss of fidelity due to cultural issues or translation errors.

5. Review the item bank and update over time.   Work conditions change, so it’s important to manage your item bank as the job requirements change. You need to regularly review the item bank and change or remove items that are no longer appropriate.

Piloting, then reviewing your trial SJT

Our 7 SJT Deployment steps

Our key steps in analyzing your SJT data