- Thread starter mlcpa51
- Start date

- Joined
- Dec 30, 2014

- Messages
- 7,374

Here is a hint: Can you list all the possible scores a candidate could get?

- Joined
- Jan 27, 2012

- Messages
- 6,293

I have a question about how this process is supposed to work. What is the definition of a tie? Does that mean all five interviewers agree, three interviewers agree, or two agree? If two agree on the highest score, are both those scores thrown out? If three agree on the lowest score, are all three thrown out. Because I don't understand how the process is supposed to work, I can't even begin to analyze this.

Here is how I would analyze the much simpler problem of a three-way tie among three interviewers without the complications of discarding two out of five results.

First, I would not make HOI's arbitrary and highly implausible assumption that the scores are completely uncorrelated. Instead, I would make the much more plausible**qualitative** assumption that the scores are highly correlated. **THEN **I would make my own arbitrary **quantitative** assumption about what highly correlated means numerically in this specific case. I shall assume that it means that the difference between high and low scores will not exceed 10, but within this range scores are uncorrelated. (On a scale of 85, this is a relatively high spread.) Why do I assume that the scores will cluster. Because three interviewers are interviewing the **SAME **candidate; presumably, the qualities of the candidate affect the scores in a related way. It is easy to change this quantitative assumption so if you don't like it you can change it.

Say the low score is x and x < 76. Then there is a 1/11 chance that another interviewer will give the same score, but the chance that both the other interviewers will give the same score is (1/11) * (1/11) = 1/121. In other words, the probability of a three way tie is a bit under 1%.

What happens if we decide after looking at the results of many interviews that my guess at clustering was too broad and a better estimate is a difference of 5 between high and low scores. Then the probability of a three-way tie is

(1/6)(1/6) = 1/36, which is under 3%. So our analysis is not very sensitive to the quantitative assumption about spread.

Notice that my estimated probability of a tie is much higher than HOI's estimate. This is because he is assuming no tendency to clustering, and I am assuming a strong tendency to clustering. Different assumptions lead to different conclusions. But also notice that both assumptions estimate a probability of an exact tie as being relatively low.

Here is how I would analyze the much simpler problem of a three-way tie among three interviewers without the complications of discarding two out of five results.

First, I would not make HOI's arbitrary and highly implausible assumption that the scores are completely uncorrelated. Instead, I would make the much more plausible

Say the low score is x and x < 76. Then there is a 1/11 chance that another interviewer will give the same score, but the chance that both the other interviewers will give the same score is (1/11) * (1/11) = 1/121. In other words, the probability of a three way tie is a bit under 1%.

What happens if we decide after looking at the results of many interviews that my guess at clustering was too broad and a better estimate is a difference of 5 between high and low scores. Then the probability of a three-way tie is

(1/6)(1/6) = 1/36, which is under 3%. So our analysis is not very sensitive to the quantitative assumption about spread.

Notice that my estimated probability of a tie is much higher than HOI's estimate. This is because he is assuming no tendency to clustering, and I am assuming a strong tendency to clustering. Different assumptions lead to different conclusions. But also notice that both assumptions estimate a probability of an exact tie as being relatively low.

Last edited:

Also remember the 85 possible points given by each interviewer to each of the two individual candidates comes from 13 questions of varying point values.

So if all five give the same score, do you throw them all out? Or if two score 80 and three score 2, do you throw them all out? Your process is not well defined. You have simply ignored my previous question.

Also remember the 85 possible points given by each interviewer to each of the two individual candidates comes from 13 questions of varying point values.

So is it the total score which defines a tie, or the individual scores for each question? Because if it is the total score, there are only 86^5 possibilities. If you assume as Halls did that there is no clustering, the probability that five scores are the same is 86(1/86^5 = 1/86^4, which is effectively zero. The probability that four scores are the same is 86(1/86^4)*85(1/86)*5 = 425/86^4, which is also effectively 0. The probability that three scores are the same is 86(1/86^3)*{85(1/86)}^2*10 = 10*85^2/86^4, which is about 0.1%. So the probability that at least three tie is far less than 2 chanes out of a thousand.

That might get you a good mark in a school room exercise. If, however, this problem has some practical application, you need to understand that the reasoning in the preceding paragraph asumes that each interviewer's score is completely random and has no relation whatsoever to the candidate being interviewed. Based on my experience of Human Resource departments, the assumption that those people make decisions by rolling dice is at least semi-plausible, but if you have competent interviewers, the scores will not be random. In that case, the probability of ties is much higher than what was estimated in the previous paragraph. To get any numerical estimate consistent with competent interviewers, you need to make some sensible assumption about the degree of clustering expected.