Writing Good Doctor Job Interview Questions. Start With A Job Analysis

85Shares

This is part of a new series focusing on how those doctors involved in the candidate selection process can help improve the outcomes of selection.

Having spent decades reviewing CVs, designing jobs. And, of course, sitting across the desk from interview candidates, I have learnt a thing or two about how to select well. So what sort of things would I teach the younger me about doctor selection if I had a chance all over again? Well. One of the things I would probably start with is how to write good doctor job interview questions.

Writing a good interview question for doctor selection involves:

First, understanding the key tasks of the actual job.
Second, developing a list of Key Result Areas. Things that are critical for the doctor in the job to be able to do.
Third, Developing clear selection criteria to reflect these Key Result Areas
And only then writing interview questions to assess candidates abilities to perform these key result areas.

Added to all this its almost essential to engage “job experts” – the doctors actually performing the job right now in the job analysis and criteria development and to think about other selection tools that can also aid you.

Job analysis and design.

Job analysis and design is a critical component in achieving good outcomes in any selection process. It involves examining the job to understand what knowledge, skills, attributes and values will result in a successful outcome. Research indicates that effective job analysis can contribute about a ffifth to the successful hiring of a candidate.

Ideally every time you advertise a position you should really review it again. However, for many doctors jobs, particularly trainee doctor jobs, which recycle every year, it may not be practical to do a full job analysis each year. And, arguably things don’t change that often to require an annual review. But it is worth considering doing this regularly, perhaps every 3 years.

The process of job analysis commences with reviewing the knowledge, skills and attitudes required for the position. This process gives you insights into what the outcomes of the job are and the key result areas.

For example, a key result area for a General Surgical Trainee might be ensuring that all surgical referrals for the team that come from the emergency department are seen and reviewed in a timely manner. This information then enables the development of selection criteria which will be the mechanism by which determinations are made about applicants’ suitability for the job as well as what the best tools, including interview questions, maybe for making such selections.

What’s The Hardest Bit of The Job?

Its often good to think about what the hardest part of the job maybe. The thing that new incumbents find challenging or the thing that can make the difference between a good performer and a not so good performer. This can often form the basis of more rigorous assessment of candidates, including reviewing their CV, referee checks, other assessments and of course the interview questions. Depending on how much time you have to interview you might even wish to allocate a fair part of the interview time to assessing this key result area, including using atypical means, such as a skills test or getting the candidate to give a presentation.

As an example, it might be decided that the hardest part of the job for a new psychiatry trainee might be being able to present a patient for an order to the mental health inquiry.

In which case, the selection panel might decide to ask candidates to come for the interview 30 minutes earlier. Candidates are then given access to a set of notes and a short video with a simulated patient and given the task of preparing how they would report to the inquiry. As the first part of the interview candidates are asked to outline the reasons why the patient requires further detaining in hospital.

There’s A Big Difference Between Competence and Performance.

Competence is what a doctor can do based what they have been trained to do so far. We can often assess this from what they say in their CV and to some extent their referee reports.

Performance is what a doctor actually does day to day.

Performance depends on competence however, it is also influenced by a host of other factors.

Obviously, a level of competence is required for any doctor job. But it is the performance that counts at the end of the day. Unfortunately, performance is often harder to assess than competence as it often calls for real-world situations.

Capability Frameworks Can Sometimes Be Useful If You Don’t Have The Time.

Many health departments have developed capability frameworks which describe the broad competencies and performance for staff. They describe the types of knowledge, skills and attributes required. Often times some of the language from these frameworks automatically populates into job descriptions within the e-recruitment system.

Every Specialty Medical College in Australia and New Zealand has defined a framework for its Fellows and Trainees. Most are based on the internationally recognized CanMEDS Framework. These can be quite useful reference documents when developing job descriptions as they tend to cover the range of capabilities that doctors need (e.g. communication, teamwork, medical expertise, leadership skills, patient advocacy etc…) And if used well you can use these frameworks to develop selection criteria which reflect where a doctor should be developmentally against these frameworks.

So for example a job description for a Resident Medical Officer might indicate that their management skills should reflect knowledge of key management tools and demonstration of effective time management. Whereas a job description for a Consultant might reflect that their management skills should demonstrate respect for the role of managers and active participation in unit leadership and management tasks, including people development and conflict resolution.

Can job analysis & design incorporate competency-based frameworks in medical trainee recruitment?

UK authors Plint and Patterson have demonstrated that we can predict progress in doctor training and job performance by conducting a thorough job analysis. In their 2010 study the authors conducted job analyses to develop selection criteria for general practice training. They then designed selection instruments and methods to evaluate candidate’s capabilities against these criteria and using a validation process to assess the extent to which selection methods provide valid predictors of progress in training or job performance.

However, one general distinction between competency frameworks and selection criteria is that competency frameworks often call for much greater numbers of criteria than which may be allowed for an individual job.

So the task if using a competency framework is to distil all this information into key criteria. Again, nothing beats a proper job analysis for this.

Enter The Job Expert.

All of this work can be made infinitesimally easier if we engage the actual job experts. Who are the job experts I hear you say? They are the people successfully doing the job right now. Which in Medicine, for example, means most of us.

I am amazed how often for trainee doctor interviews the panel is comprised of people who have either never done the actually trainee job, not worked closely with trainees or have been a trainee once 2 or more decades ago.

Having such a panel for trainee selection is just giving yourself an automatic handicap from the outset.

Job Experts on selection panels is considered best practice in many industries. But not medicine it seems.

Incorporating at least one doctor on the panel who is currently in the role can save you time in developing the right selection criteria, the right selection approach, the right questions and they generally also have a gut feel for which candidates will not be good fits.

So why is there such resistance to including trainee doctors on trainee selection panels?

Reasons For Resistance.

Possibly the main reason for resistance is tradition and a lack of understanding of the additional value a job expert brings to the table in selection.

Another problem can be that often trainee selection panels are hotly contested with lots of stakeholders (hospitals, training directors, colleges) wanting to ensure that their input is considered. So often the trainee on the panel can be jettisoned if there are already too many players.

Finally, it may be difficult for trainee themselves to participate in selection. They may feel that they are judging their peers or that they have not yet “earned the right.”

So What About Doctor Job Interview Questions?

Most recruitment manuals or training programs will suggest that interview questions should fristly be developed based on the selection criteria and then wherever possible put in a behaviourally-oriented way as a Past Behaviour Question (PBQ).

PBQs are questions which ask for examples of past behaviour. They tend to be recommended by HR exerts over “situational questions” (SQs), i.e. questions which ask an applicant what they would do in a particular situation.

The reason that PBQs are recommended is based on the idea that past behaviour is a better predictor of future performance. And there is some research that has demonstrated that PBQs may be more discriminative than SQs and that candidates find it harder to manage impressions (fake responses) in PBQs.

The aim of PBQs is for the applicant to demonstrate through past behaviour that they are able to demonstrate the selection criteria.

When asking about behaviours, interviewers can note applicant responses according to the CAR approach:

Context: what the situation was.
Action: what the candidate actually did (as opposed to what others did).
Result: what the outcome(s) of the candidate’s action(s) were.

Lets take an example of such an approach to developing an interview question.

Lets say that it is determined that one of the essential criteria for an Anaesthetic Training post is that

The trainee demonstrates the ability to practice in a safe manner in the operating theatre and recovery.

A behaviourally oriented question could then be developed around this question, as such

“Please describe a time when you identified a safety issue in the operating theatre or recovery, or another similar setting. What was the situation (context), what was your involvement (action) and what were the outcomes (result)?”

A Problem With The Behaviour Approch?

The problem with PBQs is that many doctors are accustomed to answering PBQs and have little difficulty providing answers. Strong candidates may even suggest examples to questions which are not put in a behavioural format. It is not absolutely certain how to discriminate between a candidate who has clearly done their research and practiced answering in such a format versus one that has not. My preference would be to reward the cadidate that bothered to prepare.

On the other hand, it is also not uncommon for some candidates to miss or avoid the requirement to provide an example. Such doctors may attempt to answer in a hypothetical way. If this occurs the person asking the question should redirect the candidate to providing an example. If the candidate is unable to provide a suitable example, then it is best to complete the question and move on to the next and rate accordingly.

The CAR approach is sometimes referred to as the STAR approach (Situation, Task, Action, Result), in which case questions are often asked along the lines of: “Describe the situation that you were in or the task you needed to accomplish.”

Is there a Role for Situational Questions or Clinical Problems in Interview?

In short Yes. whilst, the PBQ has come to be considered the gold standard approach to selection interviewing based particularly upon evidence accumulated from a range of studies in the 1980s and 1990s. However, recently Levashina et al (2014) re-examined some of the commonly held beliefs in relation to selection interviewing. This research indicates that the difference between SQs and PBQs may not be as dramatic as previously thought.

Selection studies have demonstrated that both PBQs and SQs in traditional panel interviews have comparable reliability and acceptability. But PBQs possibly have less “fakeability” and higher predictive validity for high-complexity jobs than SQs.

In particular, a mix of SQs and PBQs within a doctor interview format may be a valid approach and an area worthy of more study. Yoshimura et al (2015) in a study of 26 medical graduates applying for specialty training reported both PBQs and SQs as equally reliable and acceptable in a multiple mini interview format.

One popular use of SQs in doctor interviews is in the format of a clinical problem in the interview. I understand why panels feel the need to introduce a clinical problem to the interview. After all, the doctor will be doing clinical work. But there are quite a few problems with this approach.

The clinical problem will inevitably only test a small amount of knowledge and understanding of whatever field of medicine we are interviewing for. So how do we account for the false positives (the candidates who got lucky and were asked a question in one of the few areas they were good in) versus the false negatives (the ones who got unlucky and hadn’t brushed up on that particular topic)?
There are actually better ways of assessing clinical knowledge. Such as actual tests of clinical knowledge.
Practically these questions tend to take up a lot of time in the interview. Especially if the candidate has to read through a complex scenario and ask clarifying questions.

Where I am okay with a clinical problem being asked is where it may be being used to set the context for assessing broader competency issues, such as a doctors approach to collaborating with others or practicing safely.

But again if being able to successfully deal with a certain type of clinical situation is deemed essential for the role perhaps its better to ask the doctor candidate themselves for an example.