Do not use unstructured interviews !!!

Warning to anyone who uses unstructured interviews!!!!

I just read a paper with a strong warning against using unstructured interviews.  They tested the method as a screening tool rather than as a formative testing tool, but I think the conclusions and insights apply across the board. 

The paper is by Jason Dana at Yale and Robyn Dawes (recently deceased) and Nathaniel Peterson at Carnegie Mellon and is published in the most recent issue of Judgment and Decision Making.  They reviewed the use of unstructured interviews in domains like job hiring, clinical diagnosis (doctor interviewing patients), and university admissions.  Their data collection looked at students making predictions of other students’ future GPA based on just biographical information or with biographical information plus an interview. 

In each of these situations, the authors report that the interviews degraded predictive performance.  Doctors were less able to diagnose a patient when they combined an interview of the patient with the medical record compared to the medical record alone.  Job hiring and college admissions were more predictive of future success based just on the application/resume than when an interview was added.

The reasons they found are actually not surprising. One real problem with unstructured interviews is that we ask different questions of each candidate/patient/applicant.  This means we are comparing apples to oranges when putting two candidates against each other.  A second major problem is confirmation bias.  For candidates where we have an initial positive impression, we ask questions where we know the answer will be positive or avoid questions that could have a negative answer.  For those where our initial impression is negative we do the reverse.  A third problem is that we think the answers to our questions are more predictive of future success than they really are.  We ask about things that really don’t matter and give points to the preferred candidate and deduct them from the others.

What is troubling about these results is that we have a false sense of confidence.  Even though unstructured interviews are completely unpredictive and even degrade prediction in most cases, we think they are helpful so we put real value in them.  We feel more confident in our selections. 

The most concerning condition, although one that probably shouldn’t surprise us, is one where they explicitly told participants that the interviewee was just repeating random answers that they were instructed to give.  The participant knew in advance that the interview was garbage.  And yet still they used the results of the interview to make their decision, had greater confidence in that decision, and even reported that the interview was helpful. 

How could the interview be helpful if the candidate was spouting pre-arranged answers?  Motivated reasoning rearing its ugly head.  As we have seen before, sports fans are more likely to bet on their favorite team, even after being told that the odds-maker significantly biased the betting line against their team.  When told that debaters were assigned to present the case opposite to what they really believe, observers still report that they think the debater believes what he is saying – in direct contradiction to what they were just told.  Our brains don’t seem to have an “ignore and forget” function.  Even when we know information is false, we can’t prevent ourselves from using it when making subsequent decisions. 

Do you find this as scary as I do?