This is Homer White's Typepad Profile.
Join Typepad and start following Homer White's activity
Homer White
Recent Activity
A further thought, while the family decides which Thanksgiving movie to watch ... . I think I can see how to resolve the issue regarding nu that I raised in Point 3. In the situation of the HP theorem the expected value of the performance of both individual solvers and group of solvers is computed with respect to a distribution nu, which is fixed -- does not depend on solver or group of solvers. For each x in the solution space X, nu(x) gives the probability that the solver (or solver-team) starts her work on alternative x. Since nu is fixed it occurs to me that the most natural interpretation, in modelling applications, is that nu describes, not on a solver's dispositions about where best to start, but rather the "manager's" dispositions about where to start the solver or team. The manager presents a problem to the individual or team and says: "Here is x, an example of a solution. As you can see, it is worth V(x). Please improve upon it as much as you can." nu(x) is simply the probability that the manager presents x as the example. Inherent ability of solvers (relative, of course, to the valuation function V) can be expressed expressed entirely through their phi-functions. For example, the solver of maximal inherent ability works like this: letting x* denote the optimal solution and x' denote the second-best solution, the solver's phi-function is p(x) = x* if x != x', p(x') = x'. If you like you could define the inherent ability of a solver as the expected value of her solution, computed relative to a uniform distribution on the set X of starting points. But of course this is distinct from the "functional" ability defined in the HP theorem, namely: expected value of the solution relative to nu, which need not be uniform. nu just needs to have full support, i.e., assign positive probability to every element of X, in order for the theorem to hold. I don't see why a formal model should assume that solvers are presented with an initial example, and presumably Singer and colleagues have moved beyond the study of HP types of models anyway. My suspicion -- that in the simulations in the Singer paper performance will be more closely correlated to the number of guesses than to C-diversity -- is unaffected by the foregoing.
I should qualify the following the following (humorously intended) advice made at the end of point 3 above: "And the practical lesson for business firms is that if there is no ability in play (i.e., no solver exercises knowledge of any association between alternatives and their values) then management should ignore considerations of diversity and hire those workers who are able to make the most guesses in the time allotted to solve the problem (and, of course, to give each worker his/her own subset of alternatives to investigate, making sure that the assigned sets are mutually disjoint)." Nothing in the HP model warrants the assumption (implicit in the above "advice" to business firms) that anyone, managers or problem-solving agents, is aware in advance or eventually becomes aware of all of the elements in the solution space X. Hence managers might not be able to pre-assign solutions for workers to investigate, and anyway the heuristics of the available problem-solvers would probably limit the solutions they would be disposed to investigate in the first place. Still, the strategy for management would be to maximize the number of approaches investigated, and that strategy might not be tightly correlated to diversity.
Well, there is a difference here. When Hong and Page introduced the valuation function V, they did so in explicitly probabalistic terms: "We consider a random value function mapping the first n integers, {1, 2, . . ., n}, into real numbers. The value of each of the n points is independently drawn according to the uniform distribution on the interval [0, 100]. Agents try to find maximal values for this random function." the probability that such a function V:{1,2, ..., n} -> [0,100] is not one-to-on is 0. So Page and Hong can assert (and should have asserted in their paper) that the statement of the HP theorem holds with probability 1. In the case of the claim that uniformity of nu, along with assumptions 0, 1 and 2, implies Assumption 3, there is no probability involved and no way that I can see to cast it in probabilistic terms. It's just a false claim.
Having had a chance during the day to read the original Hong and Page article from 2004 (where the Hong-Page "Diversity Trumps Ability" theorem -- HP Theorem for short -- is proved) and the sections of Page's 2007 popular book The Difference, where the HP theorem and related computational models are discussed, I can offer a few preliminary remarks. 1. I agree with Thompson and the many previous commenters above who see the HP theorem as unsurprising. As we can see from the proof of the HP theorem, the way one gets the "diverse" group to win is to take so large a sample of solvers that the diverse group will, with probability approaching 1, include the high-ability solvers plus enough additional solvers to guarantee that the "diverse" group explores every alternative, thus arriving at the optimal solution. But it will often be the case that the high-ability solvers are the ones getting the group most of the way to that solution. 2. Also -- regarding Page's casual inference from the HP theorem that firms should hire for diversity rather than on the basis of some "crude" measure of ability -- Thompson's remarks about the asymptotic character of the HP theorem are on point. In practical situations where the cost of a group is an reasonably-quickly increasing function of its size (as when you have to pay each team member a salary ...) and there exists a solver with very high ability in comparison to all the other solvers, then it could very well make sense for management to hire the one super-solver and accept the possibility of a sub-optimal solution rather than to hire an army of additional team-members simply to guarantee a slightly-better optimal solution. 3. The computational models assume that each element of the solution space X is equally likely to be the initial solution tried, whether by an individual or by a group. But if this is the case then it's difficult to see how the computational models can be regarded as modeling situations where ability is actually being exercised. More precisely: the only plausible scenario in which I am equally likely to pick any element of X as my initial solution is the one in which in which for any pair of alternatives x1 and x2, I am unable to come up with some rational argument as to why x2 might have a higher value than x2. So in practice I'm just a random guesser, and the more alternatives I try out the better I can expect to do. Singer's simulations convincingly show that the C-diversity of a group is more strongly associated with its performance than the HP-diversity of the group is, but I rather suspect that if he had recorded the number of guesses the groups made then he would have found that the number of guesses is even more strongly associated with performance than C-diversity is. And the practical lesson for business firms is that if there is no ability in play (i.e., no solver exercises knowledge of any association between alternatives and their values) then management should ignore considerations of diversity and hire those workers who are able to make the most guesses in the time allotted to solve the problem (and, of course, to give each worker his/her own subset of alternatives to investigate, making sure that the assigned sets are mutually disjoint). 4. This remark is for those who have read the 2004 Hong and Page paper. It contains an error that no one has, so far as I can tell, pointed out. At the top of page 16388 the authors state Assumption 3 (Uniqueness Assumption) of the HP theorem, and immediately say: "Let nu be the uniform distribution. If the value function V is one to one, then the uniqueness assumption is satisfied." Here vu refers to the distribution of initial solution. (The authors stipulate that all solvers have the same probability distribution for which alternative x in the solution space X they will check first.) However, the authors’ inference, as quoted, is unsound. It is quite easy to come up with a solution space X, a 1-1 valuation function V, and a set of solvers Phi that satisfy assumptions 0, 1 and 2 of the theorem and are such that all solvers in Phi employ the uniform distribution nu (i.e, are equally likely to take any alternative as the initial solution) but where multiple solvers are tied for highest expected performance. The following is an example: * Let the solution space X be the set {a, b, c, d} * Let the 1-1 valuation function V: X -> [0,1] be defined as follows: * V(a) = 0 * V(b) = 0.1 * V(c) = 2/3 * V(d) = 1 * Let the solver-space Phi consist of three solvers: * Solver p1 works as follows: p1(a) = a, p1(b) = d, p1(c) = d, p1(d) = d * Solver p2 works as follows: p2(a) = c, p2(b) = c, p2(c) = c, p2(d) = d * Solver p3 works as follows: p3(a) = d, p3(b) = b, p3(c) = b, p3(d) = d Then Assumptions 0, 1 and 2 of the HP Theorem are satisfied, but the expected performance for the three solvers (using the uniform distribution nu) is as follows: • E(p1) = 0.25 * (0 + 1 + 1 + 1) = 0.75 • E(p2) = 0.25 * (2/3 + 2/3 + 2/3 + 1) = 0.75 • E(p3) = 0.25 * (1 + 0.1 + 0.1 + 1) = 0.55 I wonder how the referees missed this. Of course this is not an error in the HP theorem itself: it’s just that the theorem applies much less widely than the authors believe it does. 5. Thompson is right about the significance (or rather, lack thereof) of the HP theorem, but Singer is correct in his criticism of her objections to the simulation results. And I agree with some other commentators that Thompson falls short in intellectual charity. It was unkind of her to make such a fuss about the authors not clearly stating the 1-1 condition on the valuation function V, especially when we see that they appear to have at least recognized the condition’s importance (see point 4 above). Thompson herself was caught in a similar minor error (see Singer’s remarks on the construction of HP-diverse sets in her computer code) and was not raked over the coals for it. And when you are sub-title your paper “An Example of the Misuse of Mathematics in the Social Sciences” aren’t you inviting your social-scientist interlocutor to deny and double down instead of honestly addressing the issues you raise? On the other hand Page’s response to Thompson in the College Fix article is similarly over-heated, and some of his remarks indicate that he may be a bit muddled and disinclined to think mathematically. (See, e.g., “These technical details aside, the mathematical fact remains that one can construct any number of sufficient conditions for diversity to either trump or defeat ability. The union of these sets constitutes a rather large portion of the space of possible instances of the model.”) Maybe there is a history here that renders Thompson’s irritation understandable at least. That’s all I’ve got for now. I’m interested to look into the other papers that Singer et al. have written, and to try out some computer simulations myself. There is something intriguing about the idea that diversity could elevate performance, even if in the end the various computer simulations that suggest its value may add little or nothing to the anecdotal evidence that gives rise in the first place to the adage that “two heads are better than one.”
I spent two hours this morning quite pleasantly, going through Thompson's original article and Singer's response: "Diversity, Not Randomness, Trumps Ability." I'm not prepared yet to render any verdicts -- I need to study some of the cited articles/pre-prints and obtain the simulation code from the authors -- but I would like to thank my old undergraduate classmate Brian Leiter in advance for posting on the topic. There is currently a circle of students at my little College who are developing a nice mix of R-programming and philosophical skills and who might be persuaded to take on some piece of this work as a Spring semester project.
Homer White is now following The Typepad Team
Nov 27, 2019