We have nearly 30 years experience of bespoke psychometric test designs.

An Associate Fellow of the British Psychological Society, Rob Williams is a Chartered Psychologist with over 25 years of experience working and designing tests. He is also the author of five psychometric test design books and has worked for the leading global psychometric test publishers including SHL, Kenexa IBM, MBTI, CAPP and SOVA Assessment.

Contact Us Below – ANY corporate or educational assessment enquiry

Bespoke psychometric test design specialities

Situational Judgement Test Designs

Here’s our ten key situational judgement test design lessons:

I – Match to job competencies

II   – Conduct a range of job analyses

III  – Probe for scenarios AND answers

IV  – Follow the ‘Rules of Item Writing”

 V   – Score both SJT answer options

VI  – Leverage a subject-matter expert trial stage

VII  – Include realism rating scale

VIII – Maximise your trial numbers

IX   – Validate

 X     Update your norms/equal opp’s analysis

                  I.           Match to job competencies

You can buy off-the-shelf SJTs which cater to your industry or which are specific to a function such as sales. However, you then lose that one bit of magic that really makes an SJT special. This is that each question can be crafted to assess critical role behaviours. Having role competencies at your disposal, better yet behavioural indicators is helpful in scenario design. Although…

                II.           Conduct a range of job analyses

You should run critical incidents role analysis to capture all the critical behaviours: defined as those that lead to successful or unsuccessful workplace outcomes. For example, a happy or an unhappy customer interaction.

With, or without, a set of up-to-date competencies for your role, you must schedule as many role analysis sessions at the start of your design process as time (and money!) will allow. 

If there are national – or international – role differences then it’s vital, particularly in the US, to document how your job analysis plan captures representative sample(s) for each role being assessed.

Logistical pros and cons will certainly dictate the balance of role analysis formats you adopt. I usually recommend a blend of face-to-face strategic interviews with senior stakeholders; a telephone-based approach for job incumbents, but running separate focus groups for their line and regional managers.

            III.           Probe interviewees (for a range of options)

A few people in your sample will provide most of the information you need. Simply must include the obligatory Pareto Law quote here and say it’s exactly 80% of what you need.

Take your time and probe these individuals carefully, going beyond the critical scenarios to explore possible correct and incorrect responses to that situation. Ideally also what a less experienced colleague may do in that situation. Then voila, your interviewee has written the SJT question for you!

             IV.           Follow the “Rules of Item Writing”

More seriously there’s lots of “rules” to writing clear and concise SJT scenarios and answer options. Too many to cover here in detail. Suffice to say that for each broken item writing rule, your SJT’s validity will suffer that little bit more.

               V.           Score both answers

When designing the scoring key/format, my preference is usually to score both the Most and the Least answers chosen. This has always worked well in terms of variance and final item selection.

              VI.           Include an SME Trial

A subject-matter expert (SME) review is vital. It must be done pre-trial so that any unsuitable scenarios are discredited before any valuable trial time is wasted on them. Do schedule in your own editorial time since you need to prepare a finessed Role Incumbent Trial version next.

Aim for 15+ subject matter experts from across the business who have different selection, managing and developing responsibilities for the role being assessed.

You can sell this process to senior managers as “the real role experts”, thereby enhancing their engagement in your entire SJT design project!

           VII.            Check scenario realism

Your trial phase gives you the opportunity to get respondents to rate how realistic the scenario/answer options are. This is invaluable at the trial data analysis phase.

         VIII.           Maximise trial numbers

Aim for 300 current role incumbents to trial your SJT. Do extend the trial period if necessary to achieve this. However, such a high number is needed to ensure that you have sufficient numbers within each subgroup (for your equal opportunities analysis).

             IX.            Validate

To set accurate cut-off scores, it’s advisable to have collected a form of criterion validity evidence. You are likely to have two samples: your trial group and your validation sample. If the world was a fair place these two would have the same subjects in. In reality, they don’t, and you need to focus on those trial participants for whom you also have job performance data.

                X.           Updated norm group

My final, key lesson is to use an enhanced norm group as soon as you can, ideally from the start. Once you have sufficient numbers of “live applicants” this is the best norm group to use and any cut-off/equal opportunities analysis can be adjusted accordingly.

Bespoke personality test designs

Bespoke personality tests have become a key tool for development. I’ve recently produced motivational, customer-centric and sales-specific tools.

  • TPI’s sales personality design
  • Leadership assessment design for TPI
  • Alison’s  

Personality strengths research

Bespoke personality tests have become a key tool for development. I’ve recently produced motivational, customer-centric and sales-specific tools to name but a few.

Psychometric consultancy specialisms

We also help our clients to answer questions, such as:

Is this psychometric test valid?

In other words, is there scientific evidence that the psychometric’s results relate to job performance?

Validity measures how accurately an assessment can predict job performance. Job performance prediction is usually demonstrated with positive correlations between psychometric test scores and the job performance criterion. Well-constructed psychometric tests have 0.25 – 0.4 positive correlations with job performance.

In the US, this validity evidence must be gathered prior to using the psychometric. Whereas in the UK, we’d recommend checking with the psychometric publisher, that valididty had been shown with multiple employee groups and across several different roles.

Our aptitude test design projects

Citibank Aptitude Test Design Example

  • Designing blended assessments for retail and call centre operations across the USA.
  • Allowing for cultural differences in Mexican operations.
  • Managing item writing team for SJTs, ability tests, biodata questions etc.

Bespoke psychometric test designs

EPSO Aptitude Test Design Project

  • Development of project management test.
  • Design of IT skills-based aptitude tests.

Assessments Design

This psychometric test design is similar in nature to personality questionnaire design, explained here. Our bespoke strengths test design process…

  1. Has dimensions key to the role being assessed.
  2. Reflect the personality, attitudinal and motivational aspects of the role-specific dimensions.
  3. Have face valid questions.
  4. Be capable of completion in 20 minutes approx.
  5. Adopt a single-stimulus question format (Likert scale).
  6. Adopt a normative format of scoring utilising a sten look-up table (for each personality scale).

Situational Strengths Test

An excellent example is CAPP’s Situational Strengths Test. This builds on the many advantages that SJTs have, being:

–Easy to administer to a large audience.

–Relatively strong validity.

–Can assess a sample of the relevant KSA’s.

–Refreshed items possible.

–More difficult to fake responses.

Bespoke VBR Example

  • 3-4 scenarios will assess each value, giving approx. 15-16 questions.
  • Provides accurate and meaningful feedback to each respondent.
  • This practical job preview has the advantage of using the same SJT format.

Bespoke psychometric test designs

Situational interview(s) comprising:

  • Values.
  • Competency; and
  • Technical questions.

Situational Simulation Exercises

  • Scenarios from job analysis used to design simulation exercises.
  • Parallel version developed to maximise exercise integrity.
  • And to minimise the risk of applicants sharing details of tools.
  • Without compromising the validity of the assessment process.

Situational Values-Based Interview

  • We propose a structured interview comprising values, competency and technical pharmacy questions
  • Collaborate with job incumbents to design technical questions/scoring guide.

A situational judgement test, or SJT, presents realistic scenarios similar to those that would be encountered when doing a particular job. Typically, candidates must identify the Best and the Worst course of action from four multiple-choice options. In other words, you must make two “judgements” about what to do and what not to do next in the “situation” presented. An alternative SJT format asks the test-taker to rank multiple-choice options in terms of effectiveness.

Personality strengths in the news

On BBC Radio 2, in February 2014, Dr Almuth McDowell referred to the benefits of situational judgement tests and one of the SJT example items developed by Rob Williams Assessment Ltd.

The same interview on SJT benefits also featured in The Times on 4th Feb, Dr Almuth McDowell, a lecturer at the University of Surrey, says psychometric testing has an important role to play, but only in conjunction with other measures, not least because it is possible to cheat. 

Additional situational judgment test tips are available in Rob Williams Assessment Ltd’s latest Career Builder article entitled “How to succeed at management interview tests”.

Strengths Assessment Development

Psychometric test examples are useful to see the type of strengths development testing approaches.

In fact, there are many many types of psychometric test examples. Let’s consider the many different strengths that a person may possess. For example, their intelligence strengths and weaknesses from the many different types of intelligence.

The Intelligence Test demonstrates the range of possible intelligence strengths. There are many types of intelligence. For example, creative intelligence and musical aptitude which we have not included in our model.

Bespoke psychometric test designs

Strengths Assessment Design

An effective competency design framework is crucial to the operation of many HR practices.

Shown below is an example of a typical competency design framework used in an assessment centre. In this case, the group exercise’s competency component of the overall assessment centre competency matrix:

– Oral Communication

How clearly and confidently the individual communicates with the group.

– Planning and Organising

How much consideration is given to planning and systematically going through the issues outlined in the scenario.

– Judgement and Decision-Making

How logically the individual makes their decisions and judges other participants’ input/comments.

– Analysis and Problem-Solving

How effectively the individual analyses the scenario’s issues and the solutions proposed.

– Finding Solutions

The creative thinking that is shown. The number and effectiveness of the ideas generated.

– Teamworking

How well the individual works with and encourages the other group exercise participants.

Strengths framework design

Having a competency framework helps an organisation to:

– Know which knowledge, skills and abilities are important for success in each role

– Measure and benchmark organisational performance

– Select, develop and performance manage employees (against the competencies associated with each role).

– Is it more cost effective to adapt an already validated model?

– How best to get buy-in across the business? Firstly, for involvement in the development stage. Secondly, to get commitment to future use of the validated framework.

– What role analysis techniques are most suitable for analysing each of the job roles?

– How to benchmark employee performance against the trial competency framework?

– How to validate the effectiveness of the draft competency framework?

Strengths-based Telephone interview design

Previous projects that have encompassed telephone interview sift designs. These include:

  • Competency-based sift design for telephone and cv-based sifts (finance sector).
  • Telephone interview script design (consultancy firm).
  • Conducting telephone interviews for the armed services.
  • Telephone interview design (manufacturing sector).

Bespoke psychometric test designs

Strengths Development – Role Analysis

There follows a Role Analysis Case Study: a US bank had two specific requirements: ensuring a representative sample by samplying sufficiently high number of people per role. Interviewees were mainly high performers; sampled across both urban and rural branches; with their offices scattered right across the U.S.

1) Strengths Telephone-based interviews

Firstly, these formed a large part of the job analysis research. The advantages of using a telephone-based approach was that a standard template could be created with a script to adhere to whilst also allowing some follow-up question flexibility.

2) Assessment Designs – Visionary interviews

Secondly, these were run on a more select basis with a range of senior managers (within each job role). A scripted template ensured that all psychologists asked the standard set of questions. Face-to-face visionary interviews were also arranged with the most senior personnel at some of the focus group sites.

3) Strengths Focus groups

Thirdly, these were conducted in several American States, situated in Central, Eastern and Western time zones. A semi-structured focus group format represented the most suitable job analysis technique to use. Six participants, drawn from each job role, were invited to attend each role-specific focus group session.

4) Strengths Role survey

In addition, a role survey was created based on the bank’s competency framework, with two questions per competency. Survey results were also used to inform the item writing process.

The final 5-10 minute survey had two banks of questions with the stems:

  • How important do you do each of the following on a daily basis? and how important are the following to your job?

Example job analysis briefings

Candidate Experience Briefing – A separate briefing detailed the client style and language (US English) needs.

The main aims from a “Candidate Experience” perspectives included:

  • Reflecting the company brand
  • Engaging job applicants
  • Providing some elements of a realistic job preview
  • Overall look and feel to be consistent

Assessment Designs – Focus Group Briefing

The focus group aims were to confirm the role profile interview data.

Firstly, focuses on how best to structure the upcoming focus group sessions. Secondly, a consideration of the outputs required. Thirdly, how individual consultants could best run their focus group sessions.

Situational Strengths Test

One company that uses this test is Lloyds pharmacy, where the CAPP Situational Strengths Test is used to recruit pharmacists.

Situational strengths tests have these advantages (in any context):

– Easy to administer to large audience.

– Relatively strong validity.

– Can assess sample of the relevant KSA’s.

– Refreshed items possible.

– More difficult to fake responses.

There is a specialised form of situational judgement test called the Situational Strengths Test. One company that uses this test is Lloyds pharmacy, where the CAPP Situational Strengths Test is used to recruit pharmacists.

Situational strengths tests have these advantages (in any context):

– Easy to administer to large audience

– Relatively strong validity

– Can assess sample of the relevant KSA’s

– Refreshed items possible

– More difficult to fake responses

Bespoke psychometric test designs

Bespoke VBR Example

  • 3-4 scenarios will assess each value, giving approx. 15-16 questions
  • Provides accurate and meaningful feedback to each respondent
  • This practical job preview has the advantage of using the same SJT format

Assessment Designs – Situational Exercises

  • Scenarios from job analysis used to design simulation exercises
  • Parallel version developed to maximise exercise integrity and minimise risk of applicants sharing details of tools and compromising the validity of the assessment process

The following situational strengths scenarios are examples only. Actual client projects always start with the most suitable level of role analysis. Then again validate the content against client requirement by choosing the most predictive scenarios and answer options.

Every scenario used also gives candidates a realistic insight into the role. Thus your applicants’s understanding of their own fit with the role. This operates in addition to the assessment component (of the same situational strengths test).

Each scenario used also gives candidates a realistic insight into the role. Thus your applicants’s understanding of their own fit with the role. This operates in addition to the assessment component (of the same situational strengths test).

Every scenario used also gives candidates a realistic insight into the role. Thus your applicants’s understanding of their own fit with the role. This operates in addition to the assessment component (of the same situational strengths test).

Aptitude test design projects

Ability tests for graduates, including problem-solving aptitude tests and abstract reasoning tests.

UKCAT test practice questions for several clients. Each project encompas

  • Verbal reasoning practice tests;
  • Numerical reasoning practice tests,
  • Abstract reasoning practice tests and
  • Decision-making practice aptitude tests.

Firstly, verbal analogies test.
Secondly, ability tests at basic, graduate and at senior managerial levels.
Thirdly, numeracy tests.
Fourthly, practice critical reasoning ability tests (for the LNAT);

Bespoke psychometric test designs

Job Preview Design

Many application processes now start online with a realistic job preview design that potential applicants take on the company website to “test” if the role still appears suitable after finding out about the realities of the role. Hence it’s a job preview but a realistic one.

Realistic role preview design

Typically, the test format is similar to an SJT in that job scenarios are presented which give the test-taker / job applicant an idea of what the job entails. Whilst the realistic job preview (RJP) is not a testing phase often there will be a more sophisticated SJT in a subsequent phase. In more sophisticated examples still graphics or video technology is used to further enhance the Realistic job preview / SJT experience for candidates.

Virtual strengths design

This are an area that Rob Williams Assessment Ltd also specialise in. This builds on our situational judgement test design and we are often asked to design both a realistic job preview and a situational judgement test at the same time.

Our approach to the design of realistic job previews has the following phases.

Step-by-step role preview design 

(1) Planning a representative sample

(2) Conducting a highly detailed  set of telephone interviews and focus groups.

(3) Item writing

(4) SJT and RJP item reviewing

(5) compile a trial version to be reviewed by 10-12 subject matter experts (SME’s) in the role

(6) Trialling the situational judgement test

(7) Statistical analysis

(8) Presentation of trial results at standard-setting meeting with core SME’s

(9) Validation and norming the situational judgement test

Bespoke psychometric test designs

Situational design Exercises

  • Scenarios from job analysis used to design simulation exercises.
  • Parallel version developed to maximise exercise integrity.
  • To minimise risk of applicants sharing details of tools.

The following situational strengths scenarios are examples only. Actual client projects always start with the most suitable level of role analysis. Then again validate the content against client requirement by choosing the most predictive scenarios and answer options.

Every scenario used also gives candidates a realistic insight into the role. Thus your applicants’s understanding of their own fit with the role. This operates in addition to the assessment component (of the same situational strengths test).

Every scenario used also gives candidates a realistic insight into the role. Thus your applicants’s understanding of their own fit with the role. This operates in addition to the assessment component (of the same situational strengths test).

Situational Strengths Research

Employers are increasingly using bespoke situational judgement tests (where the candidate is presented with scenarios and asked to select the best and the worst thing to do next) as a way to learn more about their character and attitudes to work.

The increase in the number of graduate courses and the career benefits of having a degree have driven a huge increase in the number of graduates. There remains a limited number of vacancies each year on graduate trainee schemes.

During the recent years of recession, the number of graduate entry roles became even more restricted; making graduate recruitment even more competitive. Microsoft, for example, received 15,000 job applications for each of its 150 graduate position in 2009. That year, the success ratio of applications to job offers was 1:100 at Microsoft.

Bespoke psychometric test designs

This has created a “bottleneck” between the high number of recent graduates and the considerably lower number who are successfully placed on graduate entry schemes. Clearly, there are severe implications of such a challenging job market for graduates.

For employers too, there is a “war on talent” to find and sign-up the best possible graduates in the marketplace. It is a recruiters’ market, however most recruiters want to recruit the top echelon of high-performing graduates who are showing the best leadership potential, the most effective ability to work in teams, the highest levels of motivation and drive etc. SJTs offer an effective means of measuring each of these abilities and attributes.

Contact

Follow us

Facebook Twitter Instagram