Faking that NASA faked the moon landing

By Stephan Lewandowsky
Professor, School of Experimental Psychology, University of Bristol
Posted on 12 September 2012
Filed under Cognition
and Klaus Oberauer

Data integrity is a central issue in all research, and internet-based data collection poses a unique set of challenges. Much attention has been devoted to that issue and procedures have been developed to safeguard against abuse. There have been numerous demonstrations that internet platforms offers a reliable and replicable means of data collection, and the practice is now widely accepted.

Nonetheless, each data set must be examined for outliers and “unusual” responses, and our recent paper on conspiracist ideation and the motivated rejection of science is no exception.

Perhaps unsurprisingly, after various unfounded accusations against us have collapsed into smithereens, critics of our work have now set their sights on the data. It has been alleged that the responses to our survey were somehow “scammed,” thereby compromising our conclusions.

Unlike the earlier baseless accusations, there is some merit in casting a critical eye on our data. Science is skepticism and our data must not be exempt from scrutiny.

As it turns out, our results withstand skeptical scrutiny. We will explain why in a series of posts that take up substantive issues that have been raised in the blogosphere in turn.

This first post deals with the identification of outliers; that is, observations that are unusual and deserve to be considered carefully.

Outlier detection and identification

Let’s begin by examining the variable of greatest interest in our paper, namely the indicators for “conspiracist ideation,” which is the propensity to endorse various theories about the world that are, to varying extents, demonstrably unfounded and absurd (there are some reasonably good criteria for what exactly constitutes a conspiracy theory but that’s not at issue here).

The full distribution of our conspiracy score (summed across the various items using a 4-point scale ) is shown in the figure below. Ignore the vertical red line for now.

For simplicity we are ignoring the space aliens for now (which formed a different indicator variable on their own), so the observations below represent the sum across 10 conspiracies (remember that the "convenience" theories involving AIDS and climate science are omitted from this indicator variable for the reasons noted in the paper).

Thus, a person who strongly disagreed with all conspiracies would score a 10, and someone who strongly agreed with them all would score a 40.

 

The figure invites several observations. First, the distribution is asymmetrical, with a longish upper tail. That is, most people tend to more or less reject conspiracies; their score falls towards the lower end of the scale.

Second there are several observations at the very top that may­—repeat may—represent aberrant observations. It is those extreme scores that critiques of our data have focused on, for example the very thorough analysis by Tom Curtis. The top two extreme data points are indeed unusual. But then again, one might (just) expect a few such extreme scores in a sample of more than 1,000 people given the shape of the distribution.

So how does one deal with this situation?

The first, and most important step is the recognition that once the data have been obtained, any identification of an observation as an "outlier", and any decision to remove a subset of observations from analysis, almost inevitably involve a subjective decision. Thus, a valuable default stance is that all data should be retained for analysis. (There may be some clear-cut exceptions but the data in the above figure do not fall within that category).

There are two ways in which data analysts can deal with outliers: One is to remove them from consideration based on some criterion. There are many candidate criteria in the literature, which we do not review here because most retain an element of subjectivity. For our analysis, we therefore elected not to remove outliers by fiat, but we instead ensured that the inclusion or exclusion of any potential outliers has no substantive effect on the results.

That is, we examined the extent to which the removal of outliers made a difference to the principal result. In the case of our study, one principal result of interest involved the negative correlation between conspiracist ideation and acceptance of science. That is, our data showed that greater endorsement of conspiracy theories is associated with a greater tendency to deny the link between HIV and AIDS, lung cancer and tobacco, or CO2 emissions and global warming.

How resilient is this result to the removal of possible outliers?

The red line in the above figure answers that question. If all observations above that line (i.e., scores 25 or greater; there were 31 of those) are removed from the analysis, the link between the latent constructs for conspiracist ideation and rejection of climate science remains highly significant (specifically, the p-value is < .001), which means that the association is highly unlikely (less than 1 in 1000) to have arisen by chance alone.

In other words, if we discard the top 3% of the data, that is those part of the data which for conceptual reasons should arouse the greatest suspicion, our conclusions remain qualitatively unchanged.

Why discard anything above 25? Why not 29 or 30 or 18?

Because a score of 25 represents a person who disagrees with half of the theories and agrees with the other half (or some equivalent mix of strongly-agreeing and strongly-disagreeing responses). Lowering the cutoff further, thereby eliminating more observations, would eventually eliminate anyone who endorsed any of the theories—and guess what, that would defeat the whole point of the study. Conversely, there is no point in raising the cutoff because we already know what happens when all data are included.

We conclude with considerable confidence that when a highly conservative criterion (scores 25 or above) for outliers is used, the principal result remains qualitatively unchanged. Conspiracist ideation predicts rejection of science—not just climate science, but also the (even stronger) and even more strongly, rejection of the link between HIV and AIDS and the link between tobacco and lung cancer. [13/9: rephrased to clarify, the 'more strongly' refers to magnitude of regression weights.]

How does the elimination of outliers relate to the notion of “scamming”, which has stimulated so much interest in our data?

The answer is both obvious and also quite subtle: The obvious part is that the two folks at the very top of the above distribution strongly endorsed virtually all conspiracy theories. If they then also strongly rejected climate science, that would arguably constitute a profile of scamming—that is, those folks may have generated responses to create the appearance that “deniers” are “conspiracy nuts” (note the quotation marks: this discussion is almost impossible to write succinctly without labels, even if they are caricatures).

Now, we have dealt with the obvious bit about the "scamming" problem by throwing out not just those two people of greatest concern, but the top 3% of the distribution—that is, anyone who might remotely look and act like a “scammer” based on their responses to 10 conspiracy theories.

Remember—none of the significant correlations in our data disappear when those people are removed from consideration.

But now to the more subtle part: How would we know that someone who endorses all conspiracy theories but none of the science is actually a scammer? We have tacitly assumed that this somehow is evidence of scamming. But on what basis? Is there more to this than intuition?

This brings us to the fascinating issue of mental models of people's behavior, which we will address in a future post.

Bookmark and Share

110 Comments


1  2  Next

Comments 1 to 50 out of 95:

  1. Cedric Backsworth at 20:39 PM on 12 September, 2012
    Most interesting. I appreciate being informed of the actual processes behind the data analysis rather than having people try to guess what went on.
  2. Fred Upholstery at 20:42 PM on 12 September, 2012
    Hear, hear. I look forward to reading more.
  3. Now, we have dealt with the obvious bit about the "scamming" problem by throwing out not just those two people of greatest concern, but the top 3% of the distribution—that is, anyone who might remotely look and act like a “scammer” based on their responses to 10 conspiracy theories.

    Isn't it correct that the "two people of greatest concern" were found in both the groups - moon landing and climate deniers?

    If so, with a total number moon landing deniers being 10 (of more than 1,000) , then taking out these two "scammers" would make the moon-landing deniers come out more equal between climate deniers and pro-science wouldn't it?

    Was this considered when deciding the preface of the title of the paper "NASA faked the moon landing - Therefore (Climate) Science is a Hoax"?
  4. Okay tlitb1, we get it. You didn't like the title of the paper. Would you kindly step back a moment and re-read the full title of the paper - at its own face value, not the title you want to see, please? Here it is:

    NASA faked the moon landing - Therefore (Climate) Science is a Hoax: An Anatomy of the Motivated Rejection of Science

    Perhaps it would be a good idea to consider the almost 60 references?
    E.g. if dodgy titles are a problem, I'm sure many skeptics must have taken a close interest in this one:
    Nattrass, N. (2010). "Still crazy after all these years: The challenge of AIDS denialism for science." AIDS and Behavior , 14 , 248{251).
    And this one:
    "Cool dudes: The denial of climate change among conservative white males in the United States." Global Environmental Change, 21 , 1163{1172.
    And this one:
    Douglas, K. M., & Sutton, R. M. (2011). "Does it take one to know one? Endorsement of conspiracy theories is influenced by personal willingness to conspire." British Journal of Social Psychology, 50 , 544{552).
    And this one:
    Boyko , M. T. (2007). "Flogging a dead norm? Newspaper coverage of anthropogenic climate change in the United States and United Kingdom from 2003 to 2006." Area,
    39 , 470{481).
    I think it's called whimsy, and that it involves a sense of humour.
  5. @4. Bluebottle at 21:25 PM on 12 September, 2012

    Thanks for iterating the most whimsically titled references.

    To impress upon me the normality of what I "should see" in the title then I think you need further separate the whimsical part of those titles and then show me that they contained a statement of fact that was not supported by the contents of the papers themselves.

    If you can then I will accept this is normal practice in pyschological science.

    And please, don't tell me the Cool dudes whimsy clearly wasn't dealing wth body temperature and so wasn't shown in the paper ;)
  6. @7. Brad Keyes at 22:12 PM on 12 September, 2012

    Yes you are right I was letting myself become sloppy, in fact I fully agree with statement in the piece above.
    (note the quotation marks: this discussion is amost impossible to write succinctly without labels, even if they are caricatures)

    I do consider the "pro-science" and "deniers" labels as caricatures really, and I use them here only as identifiers inferred from the context of the paper being discussed. I should be more consistent with the quotes.

    The expansive debate about how people can be defined as "pro" science, and a "denier" of climate is so huge that, at least on this topic, I think it should be set aside by all. ;)
  7. I think Lewandowsky is perfectly justified in retaining his outliers. Imagine someone who, when asked by an interviewer to explain his apparently eccentric responses, replies: “I make a point of believing any conspiracy theory I come across”. You might find his attitude eccentric, and a nuisance for social scientists exploring views held by a tiny minority of the population, but I see no a priori reason for deleting it. What right has one to bar somone from the sample simply because he’s an awkward cuss?
    The problem comes from the nature of the online survey. Surveys based on face-to-face interviews derive their validity from nothing more solid than their resemblance to the common situation in everyday life of being stopped in the street by a stranger who asks you the way. It is assumed that you would no more lie to an interviewer than you would to someone who asks the way to the post office. (Though I suppose you might be tempted, if he added “Is it to the left or to the right? You are not allowed to answer ‘don’t know’”).
    The online survey is not like that. It’s more like filling in your tax form, but with no danger of being found out and punished for giving a false answer.
  8. OK, you've got my attention.

    The fundamental outstanding question is of course how your analysis and Tom's give such different results when excluding the two extreme outliers. I presume the difference comes from the method you are using. If you can explain that to a non-specialist, then you will have dramatically strengthened your case in my eyes. I am willing to learn.
    Moderator Response: Good question. Stand by for more. There are a few more posts to come on this issue. It's non-trivial and deserves to be considered carefully. SL
  9. Oddly enough, throughout this whole kerfuffle over the paper, the fact that there is a demonstrable correlation between acceptance of certain conspiracy theories and denial of AGW is only the *secondary* conclusion of the paper. The primary conclusion of the paper is, and I quote from the paper's abstract (highlight is mine):

    "Paralleling previous work, we find that endorsement of a laissez-faire conception of free-market economics predicts rejection of climate science (r ~ .80 between latent constructs). Endorsement of the free market also predicted the rejection of other established scienti c ndings, such as the facts that HIV causes AIDS and that smoking causes lung cancer. We additionally show that endorsement of a cluster of conspiracy theories (e.g., that the CIA killed Martin-Luther King or that NASA faked the moon landing) predicts rejection of climate science as well as the rejection of other scienti c fi ndings, above and beyond endorsement of laissez-faire free markets. This provides empirical con rmation of previous suggestions that conspiracist ideation contributes to the rejection of science. Acceptance of science, by contrast, was strongly associated with the perception of a consensus among scientists."

    But all the contrarians have latched on to the conspiracy theory part of it, when the elephant in the room is that it's political/economic ideology that is at the heart of the matter. I wonder why that is? Oh, look, there goes a squirrel! :-)
  10. IEHO you have yet to confront the TISSIWNE problem (hint the start is there is some and the end is I will not eat) problem that conspiracies differ in the degree to which they diverge from reality, and someone trying to game the survey would have limits.

    To give a NASA example there is a cohort of climate science deniers who are rocket jocks, and work for, have worked for, or want to work for NASA and are deeply invested in the Apollo success. Perhaps the pattern of rejectionism would be interesting.
  11. @- Steve Metzler
    "Oddly enough, throughout this whole kerfuffle over the paper, the fact that there is a demonstrable correlation between acceptance of certain conspiracy theories and denial of AGW is only the *secondary* conclusion of the paper. ...
    But all the contrarians have latched on to the conspiracy theory part of it, when the elephant in the room is that it's political/economic ideology that is at the heart of the matter. I wonder why that is? "

    Perhaps it is because the many who reject the mainstream findings of climate science do so for reasons linked to their political/economic ideology and resent the implication that their refusal to accept AGW as a significant scientific finding is linked to a conspiracy belief they also regard as foolish or mistaken.
    Foolish and mistaken in a way that they would never consider that their belief in free market economics to be.

    However the problem is that acceptance or rejection of scientific findings can never be validated by an ideological or political dogma, the credibility of scientific knowledge is not correlated with its political implications.
    Although those who clearly judge science by such political criteria have to re-frame the science as a political program.
  12. izen@13
    You make an incontestable point about scientific findings.
    But if those who reject the mainstream findings of climate science resent the implication that their refusal to accept AGW is linked to a conspiracy belief that they also regard as dopey or wrong, how come they are so often silent - or perhaps tolerant - about the company they keep? You must have noticed that, say Monckton, is self-evidently a conspiracist/birther/one-world Government. I single him out only for his prominence - my observation is based on the obviously widespread and persistent claims of AGW being a fraud, a hoax, etc. Regardless of ideology, I fully accept that there are many rational, intelligent, well-informed people with serious misgivings about some of the policy responses being suggested; what I can't accept is that so few speak out strongly and consistently against their many fellow-travellers who, it seems, really do think there's a global conspiracy.
  13. SM@ 11: "But all the contrarians have latched on to the conspiracy theory part of it, when the elephant in the room is that it's political/economic ideology that is at the heart of the matter.

    That could only be true if the possibility of 'ideology' being a substantial factor in the holding with consensus science had been tested for. On the assumption that the survey canvassed the beliefs of a lay audience, it is necessarily true that 'ideology' is a component of holding with consensus science; the mechanism of such a connection being 'trust'.

    Moreover, a distinction needs to be made between 'science' and 'consensus science'. A rejection of the latter is not necessarily a rejection of the former. For instance, the survey does not test for a rejection of the scientific method -- this much is presupposed by the analysis and subsequent commentary.

    Returning to the subject of 'ideology' and assent/dissent to consensus science, the putative consensus is broad and not exclusively 'scientific'. The material scientific basis and the policy imperatives are routinely confused, and are interchangeable in many of wider debates about policy -- especially economic policy.

    Therefore, it is not unreasonable to again suggest that there is an 'ideological' component to both assent and dissent to consensus science, but that the survey doesn't adequately take account of this, and reproduces the confusion about the object of 'consensus science' --- science vs policy. A corollary of the apparent correspondence of laissez-faire economics and the phenomenon of climate change scepticism might be an equivalent connection between an 'ideological' preference for economic and political institutions that *seem* to be the consequence of consensus climate science, and 'consensus science', but which might be only connected through 'ideology'.

    There are precedents in the history of environmental politics which reflect this confusion. For instance, Garett Hardin's influential 1969 Essay 'The tragedy of the Commons' makes an argument for the abolition of the common ownership of property, on the basis that it allows 'free riders' to over-exploit natural resources. However, the same argument is made for the control of all resources as if they were a commons -- i.e. that institutions of private property are not sufficient to ensure the protection of natural resources. This is a contradiction within environmental politics which has not been fully resolved, and has led to disagreements with the environmental movement about the best approach to reducing carbon emissions through policy, for instance through carbon-trading vs rationing.

    In summary, the survey's attempt to model 'ideology' is naive at best, and inadequate to the task of meaningfully reducing the complexities of 'ideology' and its interaction with material science to draw statistical relationships between groups of beliefs. This seems to reflect the authors' own ideological prejudices and preferences for environmental politics, which are a matter of record.
  14. First, using a spread sheet, and taking the correlation between the mean of CY question scores excluding CYAids and CYClimateChange and the mean of CC questions plus CauseCO2, I find the following linear correlations:

    -0.139 using all 1145 responses;
    -0.11 excluding just the two most suspect responses; and
    -0.074 for all responses equal to or less than 2.0833 (25/12).

    You report a pairwise correlation between the CC and CY latent constructs of -0.197 in your paper, but do not report the pairwise correlation after discarding all responses with a cumulative CY score greater than 25. If it declines by 45% as do the pairwise correlations between mean responses after excluding the equivalent responses, or by 20% as do the pairwise correlations above after removing the two most suspect responses, then that is a significant difference in your result, even if it does not greatly alter the statistical significance of your result. Therefore I need to ask, what was the pairwise correlation between the conspiracy and pairwise latent constructs in your robustness test?

    Second, excluding CYAids and CYClimateChange, you have 12 CY questions, meaning a response which contained as many scores of 2 as of 3, and as many scores of 1 as of 4 would sum to 30, not 25 as you have it. Excluding all scores greater than 25 excludes all scores with a mean response greater than 2.0833 (as noted above).

    Thirdly, when all CY scores with a mean greater than 2.0833 are excluded, I retain just 1,055 responses, meaning that 8% have been excluded, not 3% as you claim above.

    Is there any reason for these discrepancies?
    Moderator Response: Tom, your analysis of the conspiracy items differs from ours in two ways. First, we looked at the sum total of 10 items, excluding the two items on space aliens, which formed a separate factor in our initial factor analysis (and also including the conspiracy theories related to the scientific propositions). Your analysis included the two space-aliens items. Both ways of looking for multivariate outliers are equally valid—as we noted, outlier identification involves many subjective decisions—and fortunately they lead to the same conclusion (see below). Second, we analyzed the data using structural equation modeling, which enables us to compute the correlations between latent variables that exclude measurement error. You seem to have simply added the items to form scales; this is the statistically less optimal procedure.

    If we do the outlier analysis for the 12-item scale including the space-aliens items, the minimum score is 12, and the maximum is 48. A value of 30 reflects moderate rejection of half the items, and moderate acceptance of the other half. If we eliminate all subjects with a value of 30 or higher, the correlation between the latent factor for conspiracist ideation and the latent factor for acceptance of climate science drops from -.197 to -.141; both values are significant at p < .001.

    The numerical reduction of the correlation is a necessary consequence of restricting the range of the conspiracist-ideation variable – this is a simple statistical phenomenon. Effectively, by excluding every subject that could, by a very generous criterion, be regarded as an outlier, we necessarily throw away many observations that are most likely valid, and artificially restrict the variance of the conspiracy variable. This necessarily restricts the covariance between that variable and any other variable, and thereby results in an underestimation of the true correlation.

    The important conclusion is that the correlation is still clearly significant, implying that the correlation does not depend on any observations that might be outliers. In other words, our conclusions are not compromised in any way by outliers.

    On a more general note: It is always tempting to eliminate observations from data that stand in the way of one’s preferred result, and one can always find justifications for such a practice: Perhaps the offending subjects tried to make fun of the experimenter, or they did not understand the instructions, or they did not pay attention, or they confused the response buttons, or they completed the questionnaire after midnight and therefore were not fully awake, or they just looked not quite right. The practice of eliminating observations by criteria defined after looking at the data is meanwhile seen very critically in the social sciences. The better practice, which we followed, is to obtain a large sample, such that potential outliers or aberrant respondents make little difference to the results, and to analyze the data with and without potentially aberrant observations and ensure that the conclusions don’t differ. K.O.
  15. A quote from page 7 of your own article: "Links were posted on 8 blogs (with a pro-science science stance but with a diverse audience); a further 5 \skeptic" (or \skeptic"-leaning) blogs were approached but none posted the link."

    Sorry, but I have learned more about today's science from skeptic (anti science?) blog http://climateaudit.org than from any of the 7 "pro" science blogs that you mention.
  16. Could you comment please on the IP address distribution among your questionnaire respondees? Were there any duplicates?
    Moderator Response: As stated clearly in the Results (p. 8), duplicates were removed.
  17. izen said:

    "However the problem is that acceptance or rejection of scientific findings can never be validated by an ideological or political dogma, the credibility of scientific knowledge is not correlated with its political implications."

    Exactly. But then why does every prominent U.S. republican candidate reject AGW outright? Maybe because if it's true, we would actually have to do something about it, and that costs money and requires inter-governmental cooperation? Free market types have historically had a very bad track record dealing with externalities, and this is, like, the biggest one there ever was.
  18. You have no valid control group. How many non-climate science blogs have people that believe in conspiracy theories?
  19. Explain how your off topic questions in your questionaire are not leading questions?

    Here is an example of what is meant by leading questions
    http://www.youtube.com/watch?v=G0ZZJXw4MTA
  20. Dr Adam Corner, cardiff uni, first wrote about this paper in the Guardian, back in July. Adam reproduced this article on his cardiff, nottingham uni backed blog Talking Climate, where Geoff and Adam had an interesting discussion in the comments.

    The blogs surveyed and links to Six articles with the surveys that were found.
    Skeptical Science claim they hosted it and detest it after the survey period.

    A concern is that some of these blogs telegraphed what was bring looked for in the article announcing the survey. As these blogs are generally anti climates sceptics, this should be a concern.

    Links from Dr Adam Corners blog.

    Actual links to the original articles.. these were the links I found:

    http://scienceblogs.com/illconsidered/2010/08/counting-your-attitudes/http://profmandia.wordpress.com/2010/08/29/opinion-survey-regarding-climate-change/http://scienceblogs.com/deltoid/2010/08/29/survey-on-attitudes-towards-cl/http://hot-topic.co.nz/questionnaire/http://tamino.wordpress.com/2010/08/28/survey-says/http://bbickmore.wordpress.com/2010/08/28/take-a-survey

    The comments of participants at the more high traffic blogs, also expressed concerns about methodology, that perhaps the authors could address.

    The 2 other blogs 'surveyed' no links yet.
    did the authors keep a copy if how the blog owners introduced the survfyto their readers. Ie Mandias was the best

    http://www.skepticalscience.com http://www.trunity.net/uuuno/blogs/
  21. "We conclude with considerable confidence that when a highly conservative criterion (scores 25 or above) for outliers is used, the principal result remains qualitatively unchanged."

    Only because you are including "climate related" conspiracies.
    Ignore climate related conspiracies, and the few outlier responses, and you see that in actual fact skeptics were marginally LESS likely to believe conspiracy theories.

    Had you added a "big oil is conspiring to suppress climate truth" conspiracy, you'd have found the "warmists" were suddenly much more conspiratorial. By including only "climate denier" conspiracies in your questionnaire your results were going to be obviously unbalanced.

    It's almost beyond belief that such a poor study could ever see the light of day.
  22. I would also like to point out that the study's results still hold if you remove *all* the respondents who answered '4' to *any* of the "conspiracy" questions.

    (By "results still hold" I mean that you still get a highly significant correlation between mean conspiracy score - excluding Climate - and the CauseCO2 variable, and a highly significant partial correlation after controlling for average "free market" score. And, yes, a very small but still significant correlation with the Moon landing question.)
  23. Interesting that it comes down to surveys to prove that catastrophic anthropogenic global warming (CAGW) is a fact. (-snip-)

    Knowledge may be a commodity (whether the knowledge is correct or not), but expertise will never be. As a paleohydrologist currently working on a Ph.D. in a physics field and writing a dissertation on an alternate conceptual model to CAGW, I don't seem to have a place in your survey.

    That's also funny to me, because from professional experience interacting with CAGW proponents, (-snip-)
    Moderator Response: Inflammatory tone snipped.
  24. Perhaps I'm not alone in wishing folks would keep their remarks at a substantive level here.

    Mistaking a different survey for the one under discussion, referring to irrelevant and stale academic scandals of the past, bringing up Gleick as a ritual whipping-boy, all these things are useless compared to the very few remarks following the standard set by Tom Curtis.

    If you're outraged but don't have the means to explain why, surely it would better to pipe down and let people with better skills articulate objections, without getting under their feet and making an incoherent mess of the discussion?

    If you do feel you have something to contribute, why sully your points with florid adjectives?

    Maybe it's a little late to change the moderation rules but a more ruthless noise filter would surely help this conversation, now that it's entered the actual meat of the matter.
  25. Dr Adam Corners blog and Guardian article is here
    http://talkingclimate.org/are-climate-sceptics-more-likely-to-be-conspiracy-theorists/
  26. "Interesting that it comes down to surveys to prove that catastrophic anthropogenic global warming (CAGW) is a fact."

    I heard Arctic sea ice and the likes of Brown et al (2012), amongst others, were doing a fine job of that.
  27. Typos !! I'm writing on a smartphone in sunlight.. detest = deleted in my earlier comment

    Do the authors have a copy of the wording that introduced the articles to blog readers, at the 2 blogs, as we do not have available the article to see now.

    Scott Mandias was suitable neutral, others wee much less so.
  28. Thomas, if history is a guide there will -never- be adequate documentation of survey sufficient to satisfy people who are not concerned with the machinery of the survey itself but instead do not like what it says. There are two threads of communication here; one is about anger, the other curiosity. One impulse can't be satisfied by elucidation, the other can.

    Lewandowsky and Oberauer have promised to lead us through a detailed description of their methods. My fear is that their level of effort devoted to this interesting explanation will be diminished to the extent that people can't stop venting anger and thus making irrelevant remarks and dishing out insults.

    Emulate Curtis. Listen, think, reply with reason.
  29. Many thanks for the response on duplicates. I am not asking about duplicate responses, I am asking about duplicate IP addresses.

    The paper says (on page 8) "duplicate responses from any IP number were eliminated (N = 71)."

    Hence it appears you allowed non-duplicated responses from the same IP address (as might occur from a shared computer in an academic lab in a university setting).

    Is each point in the survey from a distinct IP address or are multiple responses from the same IP address part of the survey?
  30. To Bowers. (-snip-) Among other skeptics, I have no issues with the facts of summer Arctic ice loss or with warming even on an averaged global scale.

    I take serious issue with the anthropogenic and catastrophic parts of any claims by experts or those who pose as experts.
    Moderator Response: Inflammatory tone snipped.
  31. zt - refer page 101 of the paper referred to on page 8:
    http://homepage.psy.utexas.edu/homepage/faculty/Gosling/reprints/AmPsy04Shouldwetrustwebstudies.pdf
  32. Yes

    Hanich made that reply to Pielke, who did nor post the survey, because he was concerned by the methodology.

    Ie
    Duplicate IP addresses, with variation in responses, were allowed..
    How do the authors know that these were not also trying to game the surveys
    How many duplicate IP addresses were allowed
    What effect does this have onthe survey

    Will the raw data, with all the responses, and all the questions answers be made available. Until now just a subset has been released after, responses removed.

    Additionally a break down of number of responses from refering blogs send essential. As somebligmuch higher traffic than others.

    If the responses come from predominantly a few high traffic blogs. Just a narrow finder if blogs have been captured.
  33. @Sou. The paper that you cite implies that multiple responses from the same computer (IP address) are included in this study.

    Gosling says: 'To avoid eliminating responses from different individuals using the same computer (e.g., roommates or people using public computer labs), we matched consecutive responses from the same IP address on several key demographic characteristics (e.g., gender, age, ethnicity) and when such a match was detected, we retained only the ?rst response.'

    Could one of the authors clearly state whether multiple responses from the same computer/IP address (albeit with changes to some responses) were included in the responses to this survey?
  34. Or just provide all the raw data, so we can all look at what was removed, and agree/discuss the reasons made.
  35. Professor Lewandowsky,
    Do you have access to the IP addresses of respondents, or are they held by kwiksurveys?

    In the latter case, are they still available, following the recent hack?

    Will you be releasing the demographics and the answers to the questions not analysed?

    Will you be releasing information allowing us to know from which blog each respondent was contacted? The latter information would be most useful for assessing whether the survey was gamed, since the possibility was discussed by commenters at certain blogs, and the suspect responses seem to cluster at certain points in the list.
  36. urls formatting error in 24#:
    these should all work.

    http://scienceblogs.com/illconsidered/2010/08/counting-your-attitudes/
    http://profmandia.wordpress.com/2010/08/29/opinion-survey-regarding-climate-change/
    http://scienceblogs.com/deltoid/2010/08/29/survey-on-attitudes-towards-cl/
    http://hottopic.co.nz/questionnaire/
    http://tamino.wordpress.com/2010/08/28/survey-says/
    http://bbickmore.wordpress.com/2010/08/28/take-a-survey
  37. "...most people don't understand anything about the scientific method."

    And we're -all- above average in our skills. :-)
  38. @- Bluebottle
    "You make an incontestable point about scientific findings.
    But if those who reject the mainstream findings of climate science resent the implication that their refusal to accept AGW is linked to a conspiracy belief that they also regard as dopey or wrong, how come they are so often silent - or perhaps tolerant - about the company they keep? "

    Given that the conspiracy theorists are almost invariably also endorsing a laissez-faire conception of free-market economics it may be tribal loyalty. Work by Altmeyer indicates that holding a 'free market' ideology strongly correlates with an authoritarian mindset. That includes deference to traditional sources of authority which preach a static set of truths as absolute certainties. It also encourages group unity and avoiding any display of internal dissension. As another poster noted the GOP does not welcome any divergence from the conformity of rejecting AGW.

    There seems to be an implication floated by recent posters that if only the website, date and IP address of all the survey answers could be examined perhaps it would be possible to detect whether the poll was 'gamed' by a colluding group to link AGW rejection and acceptance of conspiracy fantasies. I see no suggestion that it could have been 'gamed' to generate the link between the endorsement of a laissez-faire conception of free-market economics and rejection of the scientific view of AGW.

    It is difficult to see this link as evidence that holding those ideological/political beliefs would enhance the ability of the individual to detect statistical errors in temperature reconstructions from proxy data or discover the flaws in the present understanding of climate sensitivity.
  39. Can you clarify your handling of multiple responses from a single IP address, as neither your article nor the above response does so.

    If you had multiple (but non-duplicate) responses from the same IP address, did you keep all of them? If not, how did you decide which one to keep?

    If you had duplicate responses from the same IP address, did you eliminate ALL of the responses or did you keep one?
  40. Brad: I had a few thousand citations under my belt before I actually took some time out to read up on philosophy of science and the scientific method. We weren't taught it. Just as you don't need to have studied grammar to be able to speak your native language fluently, you don't need to have studied philosophy of science to do it. (Caveat: This does depend somewhat upon the field however. I'm guessing that the closer you get to the boundaries of science, the more important method becomes.)

    The issue gets messier the more philosophy of science you read. I presume from your terminology that you are thinking of the scientific method in terms of Popper? Are you aware of Feyerabend's "Beyond Method"? If Feyerabend is right, then the scientific method may even hinder progress. I actually don't agree with him: I find Lakatos much more plausible. However even Lakatos is in my view inadequate - science is in practice clearly a social endeavour, therefore to understand how science has functioned in the past it seems to me that a sociology of science is also required. The field exists, but I have yet to find time to familiarise myself with it.
  41. Further to my post yesterday, excluding all responses greater than 30 excludes 1.6% of responses; and yields a correlation between mean CC responses (including CauseCO2) and Mean CY responses (excluding CYAids and CYClimateChange) of -0.087, approx 35% less than the full data set. You need to excluded all responses with with a summed CY score greater than 28 to exclude 3% of the data.

    Could you please clarify the cut off criteria used, the exact number of responses excluded and, of course, the pair wise correlation between the latent climate change construct and the latent conspiracy theory construct as obtained in your test for robustness?
    Moderator Response: Closed hashtag.
  42. izen @45
    Thanks for your thoughtful response, I'll consider it more. But this sentence took my eye:
    "There seems to be an implication floated by recent posters that if only the website, date and IP address of all the survey answers could be examined perhaps it would be possible to detect whether the poll was 'gamed' by a colluding group to link AGW rejection and acceptance of conspiracy fantasies."
    Indeed. It seems to me that if all this furious digging for an imagined conspiracy by Lewandowsky et al reveals anything it is a concerted attempt by a colluding group to "game" public perceptions of the outcome of the survey. As some posters here have said, it wasn't so much the findings as the loud publicity they received that set the pack running.
    yes, there's a legitimate discussion to be had about the entrails of the survey and analysis. But those harassing the messenger won't change the message, which plainly hits a raw nerve. They simply protest too much. IMHO, this was a good opportunity - but now a missed opportunity - to distance themselves vigorously from the company of the conspiracy-minded who falsely or mistakenly profess real scepticism. You know what they say: Qui cum canibus concumbunt cum pulicibus surgent
  43. The "message" only hits a nerve because it is undermined by the authors methods, and sources of data.

    The papers goal was clearly stated from the beginning - to try to associate skeptics with rejection of science.

    1. The running head of the paper is "Motivated Rejection of Science."

    2. Here is Hanich's explanation of the purpose:

    Hanich to Pielke,

    the rationale behind the survey is to draw linkages between attitudes to climate science and other scientific propositions (eg HIV/AIDS) and to look at what scepticism might mean (in terms of endorsing a variety of propositions made in the media).


    3. And here is the name of the paper:

    "NASA faked the moon landing|Therefore (Climate) Science is a Hoax: An Anatomy of the Motivated Rejection of Science"


    The papers intent is extremely clear - to link skeptic's with the rejection of climate science.

    Yet the data used to support the authors claims and conclusions about climate skeptics was all from survey data collected entirely thru strongly pro-AGW sites.

    The authors claim these strongly pro-AGW sites have a "diverse audience" while offering nothing to support this important claim.

    And the data they collected strongly evidences that bias, with the small relative share of responses identified as "skeptical."

    Collecting information used to support claims about skeptics thoughts about, and alleged rejection of, climate science thru strongly pro-AGW sites, and nothing from those with a large share of skeptic members, makes little or no sense.

    I would ask the authors why they did not include one of the largest of all the climate sites - Watts Up With That. Their readership is largely skeptic based and can, and are currently showing that they will, provide the data regarding their views...

    And for the pro-AGW viewpoint, why wasn't Climate Progress selected? Probably the largest reach of all climate science oriented sites.
  44. (-snip-)

    (-snip-)

    (-snip-)

    (-snip-)

    (-snip-)

    And as a result the message is increasing being considered irrelevant by the "masses."

    (-snip-)

    (-snip-)

    :-)
    Moderator Response: Multiple instances of sloganeering snipped.
  45. @53 & @54
    Again it is asserted that the authors' intentions are known and obvious. You cannot know what is in someone else's mind and what may seem crystal clear to you may be utterly incorrect.

    I will observe also that I have searched the paper in vain for shrill puffery, heated bombast, vitriol and tawdry sloganeering. I have, however, seen ample evidence of that elsewhere in much of the commentary on this paper.
  46. bluebottle - here you go:

    NASA faked the moon landing|Therefore (Climate) Science is a Hoax: An Anatomy of the Motivated Rejection of Science


    The very title of the paper - which shows a conclusion supported by the very thinnest of threads, and vaporates altogether if you take away the suspect answers, but one that makes excellent headlines nonetheless.

    Now take a look at the media and how it is being used.

    And the authors intentions are clearly stated, in the paper, in the email to Pielke etc. I posted them above. Their intention was to report on skeptics "attitudes towards" science and their "motivated rejection of science" ... they told us they intended to look for conspiracy ideology and free-market beliefs of these skeptics as well.

    I would also submit the author's writings right here in this blog - the commentary and response regarding criticism this paper.

    (-snip-)

    In my opinion, supported by several recent stories I've seen, this type response hurts all sides of the climate science discussion.

    The public has limited attention spans these days - and when the scientists act as advocates as part of their scientific work, the public stops taking the science part seriously.

    And again, when that happens - when the public loses interest - science, no matter what side you're on, loses.
    Moderator Response: Sloganeering snipped.
  47. (-snip-)

    (-snip-)

    (-snip-)

    (-snip-)

    (-snip-)

    You be the judge.

    Civility and collegiality, respecting opinions of others even if you disagree, presenting your research and encouraging challenge and review ... that used to be the hallmarks of science.
    Moderator Response: Repetitive sloganeering snipped.
  48. Brandon Shollenberger at 12:15 PM on 13 September, 2012
    Tom Curtis, I can answer a couple of your questions/concerns. First, the reason you came up with 12 conspiracy theories to use rather than 10 is Lewandowsky omitted the theories relating to aliens. If you omit them too, I believe the numbers will line up for you.

    Second, the issue you raised about the drop in correlation is true for the author's results as well as their latent variables are close to what you did. The difference is before the values from each column get added together, they're scaled by some constant. That, plus the fact that scaling can be changed, means the drop in their correlation will likely be a bit smaller, but it will certainly be present.
  49. @Tom Curtis, re the 3%, if you still haven't picked up on it, the article talks about 10 CY items and lists those not counted for the purpose of this particular exercise (ie HIV, the two space alien items, climate change). (Don't know if that makes a difference to how you frame your question or not.)
  50. A Scott @ 53.

    Um, the argument you make at point 2 is self-defeating.

    Hanich's explanation to Pielke is clearly about 'drawing a link' between attitudes to climate change and science generally. But what that link was, and hence the title of the paper, did not come until after the survey had made apparent what those linkages are.

    I suspect the reason you are arguing that the whole question of identifying any linkage is itself inherently tendentious is that you also automatically assume that there will be a strong positive link between AGW 'skepticism' and rejection of other aspects of science (an assumption that will likely be shared - even if only semi-consciously acknowledged - by a majority of those on both sides who've been involved in the debate for some time, I suggest.)

    This somehow renders the whole process of identifying any such a link inherently unfair to 'skeptics'! I suggest it was also likely a strong motivator in the persistent rejection of participation in the survey by 'skeptic' blogs.

    It was, after all, entirely possible - but perhaps never likely! - that a survey of the attitudes of both sides of the debate would not have revealed such linkages - between 'skepticism', 'Libertarian' ideology, conspiracist ideation, and more general science rejection - at all. But because the whole concept of asking the question is unfair, the result must be, well, cheating.

    As has been pointed out extensively, whatever one might think of the original results, that claimed positive linkage could scarcely have been more directly confirmed than by the subsequent 'skeptic' reaction to this paper.

1  2  Next

Comments Policy

Post a Comment

You need to be logged in to post a comment. Login via the left margin or register a new account.