This is David Rose's Typepad Profile.
Join Typepad and start following David Rose's activity
Join Now!
Already a member? Sign In
David Rose
Recent Activity
Readers may be interested in a recent post by Gregg Caruso over at Psychology Today. Gregg does a nice job of summarizing some recent work on folk judgments of free will when confronted with cases of perfect neuro-prediction. Interested readers can find the post here. Continue reading
Posted Oct 27, 2015 at Experimental Philosophy
How do ordinary material objects persist? For instance, if a rock is smashed to pieces with a hammer, does it survive the smashing? Or, if a rock is hit with a hammer and chipped, does the rock survive? What about a rowboat that has had all of its original parts... Continue reading
Posted Sep 19, 2014 at Experimental Philosophy
Hi Chandra, I am aware that the standard SEM practice is to test a theory derived model and fiddle with it until you find one that models the data well and is still “consistent” with your theory. The model you presented in the M&L paper was presumably a model that accurately reflected your theory. We have shown that this model undermines the theory (since it does not fit the data). Now, I think you may well say that there is a family of models that are consistent with your theory and that in order to truly undermine the DSCA we’d have to undermine all of those possible models (since there may still be a model out there, within that family, that correctly models the data). But here I think that there is a bit of a disconnect between what the theory says/predicts apriori and what one will, upon gathering the evidence, say is compatible with the theory. This last bit seems to me to be fairly post hoc, and it seems to be such that the apriori theory is alternating some of its assumptions in the course of this process to try to hang onto a semblance of the original theory. So, I think that you may say that your theory is still viable, but I think that in doing so you are changing the theory. Now, you’re right to point out that I do favor a data-driven approach, but notice that this is not to say that I do not think apriori theorizing is inappropriate (I mentioned in an earlier post that it was appropriate). But, I do think that one good thing about the data driven approach, especially like that of Tetrad, is that it can help one better understand the space of theoretical possibilities. Tetrad can show us that other models may fit the data much better than the ones that are “consistent” with our theory and can lead us to think about the target problem in new ways—ways that usually lead us away from our original theory. I know that you mentioned that you’d like to engage this discussion a bit more in a few days and so I won’t go on about it now. But, I did want to point out that I do think that either (a) the model you presented in the M&L paper is wrong since it reflects the underlying theory (and seems to me that you take it to be the “best” representative of the theory since it is the one you chose from the “family” and as you say in the paper, it is the one that fits the data near perfectly. So, I take it that you did choose this one since it is the “best” representative.) or (b) the theory is still potentially correct since there might still be other models that could be cherry picked from the space of models that would be “consistent” with the underlying theory (and this would just be to admit that you cherry-picked the wrong one in the M&L paper). Again, if you take this latter option, I think you are revising the theory as you go along to accommodate the evidence And if you are doing this, then the data is driving the theory and the theory is changing. But, most importantly, if you are interested in cherry-picking models that are consistent with the theory, notice that Tetrad also cherry picks models (it finds the model that fits the data best) and on this score Tetrad picks a much better model—one that is inconsistent with your theory. So, at least for this data, Tetrad picks a model that is inconsistent with your proposal. You may be able to concoct models that exceed the .05 threshold, but they will still be inferior to the model Tetrad found (and note that if the best model was one that had an edge from Att to Gen that Tetrad would have modeled this and so even if we find that a model like this fits the data, Tetrad will still have found a better one). And finally, even if the new model you propose fits, it must do so without violating Markov and Faithfulness, and I doubt that it will meet these requirements. So, there may be models out there that are consistent with your theory, but I don’t think that they will be found in this data. Best, David
Hi Chandra, You mentioned that I had read a paper where you mentioned a version of the Deep Self View that predicts the model that you recently posted. Are you referring to your recent paper on manipulation cases? Thanks, David
Hi Chandra, Thanks so much for your comments. I have a few quick responses: (1) “Claim 2: The positive sub-model associated with the DSCA does not fit the data. This too is simply incorrect. A significant chi-square simply cannot be used in the way you propose to reject a model. A chi-square test is unreliable when sample size is large (i.e., >200; see the Kline text you cite, page 200) so established norms in the field are to use other fit indices, such as GFI and NNFI, which both show the positive sub-model is consistent with the data. There are also lots of problems with breaking a holistically fit (maximum likelihood estimated) and tested SEM model into parts (with resulting loss in degrees of freedom) and insisting on good fit for every part, but I will leave that issue alone for now.” You say that a chi-square is unreliable when the sample size is large. Your sample size was 240, do you consider this to be large enough to render the chi-square tests of the sub-models unreliable? More importantly though when a model has good overall fit it is both useful and commonplace to investigate *why* a model has good overall fit. The reason is because for any model I can dump in all sorts of “noise variables” that are just uncorrelated Gaussians and get a good p-value. So, looking at sub-models can be a useful way of seeing why the overall model fits well. Additionally, there is no “correct” method for analyzing sub-models (chi-square tests are perfectly apt but so are BIC and AIC scores). Again, what’s really important is figuring out why a model fits well, and by looking at the various sub-models we can look at what parts are fitting the data well and what parts aren’t. (2) (a) “But there is an even more serious problem here. You say you use two assumptions, Markov and Faithfulness, to infer certain probabilistic relations that need to hold among the variables in the positive sub-model, and then argue these probabilistic relations fail to obtain. However, you failed to mention a third critical assumption – Causal Sufficiency.” (b) “Moreover Tetrad is certainly controversial, with many serious thinkers skeptical of it, especially in terms of practical applications. For example, the Causal Sufficiency assumption or other assumptions may not be met in actual empirical data sets, and if the theorist stipulates these assumptions are met, Tetrad will misrepresent actual causal structure, which is exactly what happened in your Tetrad analysis.” The suggestion that algorithms in Tetrad do not assume casual sufficiency is surely not true. Causal sufficiency is assumed by GES and PC (but there are some algorithms in Tetrad that do not assume this e.g., BIC, FCI). So, I don’t see how the concern about causal sufficiency threatens the Tetrad project as you seem to suggest in 2b above. Relevant to 2a though, is that the disagreements between your model and the GES model are not due to causal sufficiency. (3) “Let me now turn to broader issues of your theoretical approach. You do not provide nearly enough context about the evidential status of the Tetrad framework from which you mount your critique. Structural equation modeling is an established method widely used in the behavioral/social sciences (PsychInfo alone showed more than 10,000 articles published over the last couple of decades). Sara and I use a very standard SEM methodology, and we execute it completely ‘by the book’, that is we carefully follow the established norms for proposing and testing an SEM model, and use SEM only to adjudicate between competing a priori hypotheses (rather than blindly searching for the best fitting model). You offer a critique based on Tetrad, a theory-free search procedure that is very much a new kid on the block. Tetrad is not established in the field by any means (I am aware of only a handful of studies in the behavioral sciences that have used it, and fewer in leading journals).” The way I understand the objection to Tetrad is that it should be considered “untrustworthy” since it does not benefit from having the long history that SEM use does in the social sciences and furthermore, there are no established rules governing its use. Linear regression has a long history and quite a bit of well-established rules to go along with it. However, linear regression is typically used for causal inference, and it is very unreliable when used for this purpose. This practice is widely accepted, but that doesn’t necessarily mean that such widespread acceptance should confer any statistical legitimacy upon it. Widespread acceptance by a community doesn’t necessarily mean that the practice is reliable. I would like to emphasize that apriori theorizing certainly does play *some* role (as you have emphasized throughout your criticism) and that Tetrad is not some sort of black box that takes data as input and spits out true models. Tetrad is a tool for understanding “holistic” structural features of the data statistics—and learning these features can be helpful in understanding the space of theoretical possibilities that are consistent with the data. Finally, Tetrad is a far better tool than standard SEM techniques and there are several good papers that show this. If you’re interested (and if any others are interested), I would recommend the following: (a) The TETRAD Project : Constraint Based Aids to Model Specification. (1998). Scheines, R., Spirtes, P., Glymour, C., Richardson, T., & Meek, C., Multivariate Behavioral Research, Vol. 33, N. 1, 65-118, & "Reply to Commentary," same issue, 165-180. (b) Causal Inference (2004). Spirtes, P., Scheines, R.,Glymour, C., Richardson, T., and Meek, C., in Handbook of Quantitative Methodology in the Social Sciences , ed. David Kaplan, Sage Publications, 447-478. (c) Using Path Diagrams as a Structural Equation Modeling Tool, (1998). Spirtes, P., Richardson, T., Meek, C., Scheines, R., and Glymour, C., Sociological Methods & Research, Vol. 27, N. 2, 182-225. Richard Scheines, Peter Spirtes, Clark Glymore, Greg Cooper, David Heckman and many others have written brilliant papers explaining why and how Tetrad (or more precisely, directed graphical search procedures) are superior to standard social science methodology. The burden of proof then is actually a bit misplaced and as I hope we’ve shown the real proof is in the pudding. Again, we appreciate your comments and criticisms. All the best, David
David Rose is now following The Typepad Team
Jun 21, 2010