A simple recipe for the manufacturing of doubt

By Klaus Oberauer
Posted on 19 September 2012
Filed under Cognition
and Stephan Lewandowsky
Professor, School of Experimental Psychology, University of Bristol

Mr. McIntyre, a self-declared expert in statistics, recently posted an ostensibly unsuccessful attempt to replicate several exploratory factor analyses in our study on the motivated rejection of (climate) science. His wordy post creates the appearance of potential problems with our analysis.

There are no such problems, and it is illustrative to examine how Mr. McIntyre manages to manufacture this erroneous impression.

Our explanation focuses on the factor analysis of the five “climate science” items as just one example, because this is the case where his re-“analysis” deviated most from our actual results.

The trick is simple when you know a bit about exploratory factor analysis (EFA). EFA serves to reduce the dimensionality in a data set. To this end, EFA represents the variance and covariance of a set of observed variables by a smaller number of latent variables (factors) that represent the variance shared among some or all observed variables.

EFA is a non-trivial analysis technique that requires considerable training to be used competently, and a full explanation is far beyond the scope of a single blog post. Suffice it to say that what EFA does is to take a bunch of variables, such as items on a questionnaire, and then replaces the multitude of items with a small number of “factors” that represent the common information that is picked up by those items. In a nutshell, EFA permits you to go from 100 items on an IQ test to a single factor that one might call “intelligence.” (It’s more nuanced than that, but that captures the essential idea for now).

One core aspect of EFA is that the researcher must decide on the number of factors to be extracted from a covariance matrix. There are several well-established criteria that guide this selection. In the case of our data, all acknowledged criteria yield the same conslusions.

For illustrative purposes we focus on the simplest and most straightforward criterion, which states one should extract factors with an eigenvalue > 1.  (If you don’t know what an eigenvalue is, that’s not a problem—all you need to know is that this quantity should be >1 for a factor to be extracted). The reason is that factors with eigenvalues < 1 represent less variance than a single variable, which negates the entire purpose of EFA, namely to represent the most important dimensions of variation in the data in an economical way.

Applied to the five “climate science” items, the first factor had an eigenvalue of 4.3, representing 86% of the variance. The second factor had an eigenvalue of only .30, representing a mere 6% of the variance. Factors are ordered by their eigenvalues, so all further factors represent even less variance. 

Our EFA of the climate items thus provides clear evidence that a single factor is sufficient to represent the largest part of the variance in the five “climate science” items.  Moreover, adding further factors with eigenvalues < 1 is counterproductive because they represent less information than the original individual items. (Remember that all acknowledged standard criteria yield the same conclusions.)

Practically, this means that people’s responses to the five questions regarding climate science were so highly correlated that they reflect, to the largest part, variability on a single dimension, namely the acceptance or rejection of climate science. The remaining variance in individual items is most likely mere measurement error.

How could Mr. McIntyre fail to reproduce our EFA?

Simple: In contravention of normal practice, he forced the analysis to extract two factors. This is obvious in his R command line:

pc=factanal(lew[,1:6],factors=2)

In this and all other EFAs posted on Mr. McIntyre’s blog, the number of factors to be extracted was chosen by fiat and without justification.

Remember, the second factor in our EFA for the climate item had an eigenvalue much below 1, and hence its extraction is nonsensical. (As it is by all other criteria as well.)

But that’s not everything.

When more than one factor is extracted, researchers can rotate factors so that each factor represents a substantial, and approximately equal, part of the variance. In R, the default rotation method, which Mr. McIntyre did not overrule, is to use Varimax rotation, which forces the factors to be uncorrelated. As a result of rotation, the variance is split about evenly among the factors extracted.

Of course, this analysis is nonsensical because there is no justification for extracting more than one factor from the set of “climate change” items.

There are two explanations for this obvious flaw in Mr. McIntyre’s re-“analysis”. Either he made a beginner’s mistake, in which case he should stop posing as an expert in statistics and take a refresher of Multivariate Analysis 101. Or else, he intentionally rigged his re-“analysis” so that it deviated from our EFA’s in the hope that no one would see through his manufacture of doubt.

Bookmark and Share

654 Comments


Prev  1  2  3  4  5  6  7  8  9  10  11  12  Next

Comments 351 to 400 out of 589:

  1. What ideology would that be?

    The ideology that "there isn't a big climate problem", of course.
  2. Have a read of this one - it has more choices - might help.
  3. @- HAS
    " I take it from your comments that you now agree that L. et al doesn't support the claims made. "

    Your 'take' is wrong.
    Until i see good evidence to the contrary I am willing to grant the authors the benefit of the doubt and accept the evidence and the claims as people with real expertise in the field will be peer reviewing this.

    @- "How might they have done better?
    Well they could have held out a proportion of their sample and tested the model developed from one half on the other. This would be standard procedure to get published in most disciplines. "

    I am not sufficiently familiar with the discipline of cognitive psychology to know if this is standard procedure, on an admittedly brief scan of the literature I see no reference to that methodology and I suspect that the use of SEM may obviate it.
    But perhaps you ARE familiar with the methodologies in this field to make a useful comment, how many papers have you published in cog-psych ?
  4. Fair enough. People like Prof Lewandowsky might have an answer then, but it's beyond my expertise to 'diagnose'. More reading might help determine where you differ from the science but not necessarily reveal the 'why'. However, discussing the science itself would take the thread off topic.  I'd suggest going to a climate science discussion website if you want to pursue this.

    If the mods are willing to indulge, I'll leave you with these suggestions.  (If not, the rest of this post will likely disappear, which is fine.)

    Since you've studied geology then you might find the AGU site helpful. Richard Alley is very good at describing climate change from a geology perspective. If you're into videos then this AGU presentation from Prof Alley is long but good.

    The Earth: The Operator's Manual series comes recommended as well - I haven't seen them myself, but I've been told Prof Alley's videos in this series are good too.

    Or just do more reading on the subject and go talk to some climate scientists.  (The above suggestions are only if you want to talk the same language as people who are familiar with climate science rather than using your own special language.  It's almost impossible to converse while you're using different definitions to the rest of the world.)
    Moderator Response: Thank you for trying.
  5. @406 - So far I've corrected your misunderstanding of one or two words (consensus and possibly conspiracy), I think.
    Only in your mind, Brad. Which is why I suggest trying to work out what other people mean by them. (You quibbled over what is meant by scientific consensus for a few pages. I didn't discuss the definition of 'conspiracy' with you AFAIK.)

    Remind me, have I misunderstood any words?
    Yes - see above, but let's just say you invent your own meanings. (You understand yourself, no doubt.)
  6. A few comments, again.
    1. Reading the paper, it isn't clear what method of factor analysis was applied, but that's a minor issue, because the use of the method was to identify the dimensionality of item clusters for input into the SEM model - there will be numerical, but not meaningful differences, between methods. A letter to the authors would suffice to clear this up.
    2. The SEM models are based on a large sample (1145), and report coefficients and various error measures. That's appropriate for this work, split-half comparisons are redundant here, because the statistical tests already convey the uncertainty.
    3. A sincere attempt to clarify questions without risking bringing the science, the authors, or the questioners into disrepute, would be to go through Simonsohn's steps. What was done, how does this fit with the research (is someone going to question Heath & Gifford now?), as well as a sincere acceptance that the questioner might be mistaken. In science, honesty is everything. Questioning someone's integrity without good cause is offensive, and failure to understand methodology without asking for clarification is not good cause.
  7. "Their assumption couldn't possibly affect my argument one way or the other."

    I guess that's true in a sense - specifically, because your argument is based on circular logic.

    Why you think that will persuade anyone who is actually skeptical is beyond me.
  8. I do agree with the view that prevailing ideologies at the individual level can act as very effective barriers to policies that would seem advisable for society at large. I agree with the general proposition that the philosophy of continuous (exponential) growth eventually becomes the philosophy of the cancer cell, and that some kind of steady-state economy will need to replace the current model of perpetual growth as a necessity.

    Some kind of deliberate growth stunting does seem to be emanating from the top, but they way it is being implemented is savage in the extreme, amounting to the most brutal siphoning of wealth to the very top from the rest of society, and the consequent creation of vast masses of dispossessed.

    Promoters of climate alarmism seem to operate from a varying mixture of two convictions: (1) that there is some real and urgent cause of alarm regarding our influence on the climate, and ( 2) that the propagation of this belief can act as a powerful catalyst for the implementation of remedial structural changes in society. Some promoters, like Mike Hulme, readily acknowledge that they are not really interested in debates about the validity of the first assumption. They think activism on the second is where all the effort should go.

    But it so happens that I am fiercely skeptical (borderline denialist, you could say) of the first assumption. And I know that my skepticism does not stem from an educational handicap to understand the scientific basis offered by the official version of things, which I have examined with some parsimony and found totally unconvincing; and I also know that my ideology is not preventing me from accepting the obvious need for revamping a crazy economic model that seems to have entered a phase of convulsive irrationality and savagery.

    So I agree with the Kahn paper to the extent it has shown that the commonly-assumed correlation between greater scientific ignorance and greater reluctance to accept climate alarmism is just flat wrong; and that in fact a slightly opposite correlation seems to be at work.

    I disagree (at least based on my own case) that ideological preconceptions are what's preventing the public from accepting your message.

    I am of the opinion that promoting a trivial or non-existent problem is always a bad solution to try to solve real ones. And that the main reason so many people reject the climate alarm message is that they find it extremely exaggerated, and they are understandably suspicious of schemes to turn carbon into a commodity in the financial markets, or of insane measures like converting massive amounts of food (corn) into fuel, supposedly in the name of environmental goodness.

    As I mentioned in my previous comment, it’s not for lack of airing of the climate message that people fail to get it. It may be in fact the opposite: that some kind of saturation point is reached beyond which further repetition of the message only increases rejection. Perhaps you don’t realize to what extent the message has become pervasive in the media, and how enormous the imbalance is between its coverage and the coverage of the skeptical view, which is effectively reduced to some internet blogs. So the reason for the public's reluctance to get the message is neither ignorance, nor pre-existing ideology (at least in my case) nor lack of coverage.

    What is it, then? Think about it.

    I suppose many of you have good intentions and wish to make society better. But climate alarmism is a very clumsy tool for that purpose if it cannot be convincingly substantiated. As it turns out, it can’t. Your dilemma is this: if you try to argue based on factual known specifics, the impression quickly builds that this case is based on a high stack of hypothesis and assumptions too speculative for comfort. If you eschew that approach and resort exclusively to arguments from authority, that will also fail to convince those who are capable of examining the available evidence on their own.

    (-snip-)
    Moderator Response: Link to a blog piece eschewing ideology over the primary scientific literature snipped.
  9. "well, I do know a bit about geological history—maybe that's why I can't take the idea of dangerous warming seriously? (Geologists are extremely "skeptical" as a group.)"

    Wow!

    What Sou said - or more importantly, what Richard Alley said.

    Geologists on the whole - whom you appear to cite as some sort of vaguely authoritative source of climate science consensus - don't know much climate science, certainly not enough to be authoritative about it, even though their field supplies some useful evidence to climate science.

    It's extremely interesting that you'll apparently highly value the consensus of geologists about matters outside of their own field - and value it over and above the consensus of practicing climate scientists about matters within their own field.

    How and why exactly do you think this leads you to a more accurate assessment of the weight of all the climate science evidence?
  10. "...which is effectively reduced to some internet blogs."

    That's not at all accurate in perhaps the most important country for climate change action - the US. A fairly significant proportion of the population get their news from outlets such as Fox News that have a highly skeptical take on climate science, and another fairly significant proportion get their news from other sources that generally bend over backwards to give the "skeptics" equal time in the interests of being "fair and balanced".

    And it's not exactly accurate in Australia where our highly visible Leader of the Opposition has talked about it incessantly for a couple of years now, and the most prominent national newspaper has seen fit to feature far more in the way of "skeptical" claims than those stemming from the scientific evidence.
  11. "I value both consensi equally, which is to say that zero = zero."

    Your own earlier comment undermines this claim.
  12. ""If you eschew that approach and resort exclusively to arguments from authority, that will also fail to convince those who are capable of examining the available evidence on their own.""

    Well, of course.

    But evidence at a simplified level is frequently presented in arguments targeted at non-scientific audiences.

    And the argument is clearly both from the evidence - for those capable of assessing it - and from authority based on that evidence + the unusually strong consensus about it after a massive peer-reviewed survey of the field - for those who are not.

    What other argument do you think can validly be presented to those incapable of assessing the evidence?

    Or as I wrote here if you don't have the skills and knowledge to evaluate the evidence and arguments yourself, what strategy for belief formation gives you the greatest chance of ending up where the weight of evidence lies?
  13. @- Brad Keyes
    "Maybe, just maybe, the human cerebral cortex functions in precisely the opposite way on this one particular issue compared to all the other issues it thinks about. Normally the more you know, the closer your opinions are to the truth, but climate change is unique: the more you know, the more you delude yourself."

    Or maybe, just maybe climate science is NOT unique and the more you know the closer your opinions are to the truth as in every other field. It would explain why increased scientific literacy correlates with an increasing percentage accepting climate science. Culminating with just about 99% of climate scientists accepting it as well as every major scientific body and institution. As is often pointed out there is a consensus.
    It is you who are proposing the hypothesis that uniquely among science fields climate consensus is the product of increasing delusion not knowledge.

    @-"Can readers nominate another example in history that might indicate the existence of such a mechanism?"

    Not in the field of science, but some would argue that it IS the defining characteristic of theology !
    {grin}
  14. I hope I can at the very least mention the name: The late David F. Noble was a science historian and a sharp critic of corporate capitalism in most of his work. Whether you agree or disagree with his views, his article "The Corporate Climate Coup" is worth reading for the perspective it offers. Readers should be allowed to judge his arguments by themselves, I imagine.
  15. Brad Keyes:

    Kahan et al.


    You keep saying that paper. I do not think it means what you think it means.
  16. @- Brad Keyes
    "Right. And decreased alarm. Kahan et al."

    Your enthusiasm for Kahan et al is meretricious. The are plenty of studies that confirm that increased educational and scientific literacy correlates with increased acceptance of climate science, including its 'alarming' aspects.
    But Kahan et al use a measure of scientific literacy in which the hardest question was how long does it take for the Earth to orbit the Sun ?
    At least the participants in the Kahan et al study seemed to find it hard, less than half got it right.
  17. "since [Monckton]'s an outspoken advocate of laws to protect natural habitats he surely falsifies the "avoidance of civic regulation... to prevent significant environmental damage" theory."

    Er, no - and not just because he makes false claims about where the weight of climate science evidence lies - claims which if translated into policy or merely emissions as usual will have quite bad impacts on all sorts of natural habitats (and which scientists say has already started for some).

    IIRC he has significant associations with - or is trotted out as an advocate by - The Heartland Institute (itself a member of the American Legislative Exchange Council), the UKIP (anti-tobacco regulation, amongst other things - although they ejected Monckton), The Heritage Foundation, CFACT, SPPI, Gina Rinehart and various mining industry associations, The Global Warming Policy Foundation, the American Enterprise Institute, the Institute for Public Affairs, Americans For Prosperity, The International Climate Science Coalition, The Galileo Movement, Senator Inhofe and various likeminded individuals...many of which have a keen interest in either (a) avoiding civic regulation, full stop, or (b) focusing on "free-market" (hence non-regulatory) solutions to environmental issues.

    Put it this way - he's far far more well known for his those kinds of views than for anything falling under the generally recognised definition of "environmentalism".

    (But even more seriously - Monckton, seriously?!? It's a very bad look to trot out someone so ... out there ... that the flippin' UKIP pushes him away in support for your claim that climate non-alarmism and environmentalism are relatively common co-beliefs.)

    And finally, and importantly, the categorisation here - of individuals, and separately of correlations between ideologies and various beliefs in groups - is not asserted to be binary either/or. It's a matter of degree, so a few counter-examples do not rebut the observed correlation.
  18. "Right. And decreased alarm."

    I'm starting to consider the possibility that it's a strange kind of performance art.

    You can't seriously think that readers are so gullible and ill-informed that they don't know that those who are the most informed and most skilled in climate science are rather concerned, can you? I mean, you aren't foolish enough to extrapolate from data points at the low and lower-middle end of the curve all the way to the top, are you?

    Or is it more important to you to repetitively assert dubious and even counter-factual generalisations regardless of the impacts they have on your credibility?

    Either way it is fascinating on some level, but the fascination is starting to wear off ;-)
  19. I don’t understand why geologists need to be ignored in discussions about a hypothesis that relies on the assumption that something quite unprecedented is taking place with temperatures. Geologists know things about the history of the planet. I look at the history of the Holocene and see warming events as warm or warmer than today’s, one of them called variously Climate Optimum, Holocene Thermal Maximum, etc, that coincided with the rise of human civilization. I also see that these warming events, long recognized and established in the scientific literature, began to be systematically minimized from the early 1990s coinciding with the new paradigm of global warming. It’s not difficult to find and see these things.

    If I wanted to worry about something related to future climate, I would turn my attention to the fact that judging by the last 4 glacial-interglacial cycles over the last 400,000 years or so, it does appear very much as if the beginning of the end of the current interglacial is near (where near is of necessity a very elastic word here). But the next glaciation does appear somewhat overdue, judging by the previous cycles. Now, what exactly are we going to do when the ice sheets begin expanding again over vast food-growing and heavily populated areas that were comparatively empty when the ice retreated some 12,000 ago – this is something that does give me some concern, even if I will be long gone when it happens. It doesn’t take a lot of hard thinking to come to the conclusion it’s going to be a very devastating event, and that the historical evidence for its impending arrival (again, a very elastic "impending") is infinitely more compelling than the supposition we are living in unprecedented temperatures and headed for catastrophic warming.
    Moderator Response:

    This is off-topic. But for edification purposes, per Tzedakis et al 2012, “glacial inception would require CO2 concentrations below preindustrial levels of 280 ppmv” (for reference, we are at about 394 right now…and climbing).

    Earlier, Tyrrell et al 2007 examined this, concluding that we have already skipped the next glacial epoch. Furthermore, Tyrrell concludes that if we continue our present fossil fuel consumption, we will skip the next 5 glacial epochs. So no glacial epochs the next million years to worry about…

    Please return to the topic of the OP.

  20. Would someone lift the needle please? The record's stuck. Must be scratched. Thanks.
  21. "I don’t understand why geologists need to be ignored..."

    I don't believe they are ignored (except when they claim expertise outside of their field).

    I'm not sure that your claim "I also see that these warming events, long recognized and established in the scientific literature, began to be systematically minimized from the early 1990s coinciding with the new paradigm of global warming." is supported by the evidence. A very similar claim was made about the MWP based on a sketch in the IPCC's first report - but that was clearly a sketch based on very geographically localised data rather than a portrayal of a hemispherical or global reconstruction which is necessary to establish an MWP in the first place. In other words the "MWP" that was "minimised" had not been established in the first place (and people making that claim also tended to ignore the fairly large confidence intervals that far back in the Hockey Stick graph...)

    Also, and perhaps in partial explanation of that lack of evidence, proxy measurements of temperatures back that far (e.g.) generally rely on much sparser evidence than (say) the original Hockey Stick, so you may want to consider (a) how large the confidence intervals are on any such graphs, (b) whether the temporal resolution is high enough to inform comparison with the rate of change of the last few decades and (c) whether they are even a hemispherical or global reconstruction at all - because if they aren't, it's difficult to use them to compare with current global temperature reconstructions.

    "But the next glaciation does appear somewhat overdue,...

    IIRC that's not true. The older conventional scientific wisdom before that recent research was that most interglacials tend to last about 12000 years so there's probably about 10 centuries before it becomes an issue.

    "It doesn’t take a lot of hard thinking to come to the conclusion it’s going to be a very devastating event..."

    IIRC a number of climate scientists generally think we've already or almost already done enough to the atmosphere to overwhelm the next glaciation period unless we pull a lot of CO2 rapidly out of the atmosphere. Others argue that we are in an interglacial that will last about 28,000 years, or even predict changes in orbital forcing that extend that to 50,000 years - even if we were to get CO2 levels back to 280ppm.

    IIRC it is a very uncommon view amongst scientists that CO2 is staving off any sort of imminent glaciation event - and they are far more concerned about the impacts of global warming in the next few centuries.
  22. "That's not what I said..."

    No, you didn't.

    Instead your earlier comment implied it - and you helpfully reiterated the relationship that implies it in that latest comment!

    Let me try it for the last time.

    "Increased scientific literacy —> increased acceptance of climate science."

    Let's ignore that Kahan et al does not demonstrate this, except in your bubble where you apparently derive "increased acceptance" from "less alarm" in a gyre of devastating circular logic.

    As I point out above, those with the most scientific literacy especially with regard to climate science are ... practicing climate scientists. And it's rather clear that they have on the whole very high "acceptance of climate science".

    "Increased scientific literacy —> decreased alarm."

    And yet as a group those practicing climate scientists clearly do NOT have "decreased alarm" but are in fact rather concerned.

    Why, it's almost like you implied that those who know the most about climate science and have the best skills to evaluate it exhibit "decreased alarm" when instead we know they are rather concerned!
  23. someone made this odd unsubstantiated assertion:

    "Geologists are extremely "skeptical" as a group"

    It's a statement without unambiguous meaning (wriggle room perhaps?) but worth thinking about.

    In my experience geologists have a rather good understanding of the serious dangers inherent in massive enhancement of greenhouse radiative forcing. Well-informed geologists understand that aspects of both uniformitarianism and catastrophism imprinted in geological evidence indicate very real dangers ahead.

    So those geologists that have an honest, informed insight into the deep past recognise that (inter alia):

    - Returning the massive amount of carbon sequestered for 100’s of millions of years, especially since the Carboniferous, back into the atmosphere in the form of CO2 will cause greatly increased surface warming and ocean acidification and sea level rise.

    - The warming effect of enhanced greenhouse forcing from massive release of sequestered fossil fuels is enhanced now compared to periods in the deep past as a result of the steadily increasing solar constant.

    - Many of the catastrophic extinctions in the deep past are associated with tectonic events and catastrophically raised greenhouse gas levels resulting in surface warming, ocean acidification, stratification and anoxia.

    - The evidence increasingly supports slow reductions in atmospheric CO2 concentrations as the major driver of reduced temperatures that forced the onset of glaciations. This applies both to the Arctic glaciations of the late Pliocene and also the onset of Antarctic glaciations 33-34 MYA. Evidence from geologists indicates that we are already well past the threshold required to support a Greenland ice sheet (280-300 ppm), and that we are pretty much committing our descendants to a very large sea level rise.

    - Geologists have also given us evidence that although melting of ice sheets may be slow if horribly persistent through time, that catastrophic ice sheet collapse can speed up sea level rise dramatically.

    and so on…an informed geological perspective into our deep and recent past is not the friend of those who reject scientific evidence in support of dodgy motivations.
  24. "But the next glaciation does appear somewhat overdue,...

    IIRC that's not true. The older conventional scientific wisdom before that recent research was that most interglacials tend to last about 12000 years so there's probably about 10 centuries before it becomes an issue.


    Interesting - thats all there is to it - time passed?

    How does the current inter-glacial data look compared to prior inter-glacial warm peaks? Here's a hint - they come to a pretty sharp spike, and rapidly descend into another long glacial period.

    Temps for the current inter-glacial peak have, contrary to almost every past inter-glacial warm peak, spike upward and have stopped - instead of the sharp drop they temps have hovered at that inter-glacial warm peak. How do you explain that?

    What about the entire "climate cycle" time frame - from glacial to and thru inter-glacial and back? They've been pretty regular for at least the past 500,000 years or more.
  25. Basically look at the strength of the various Milankovitch forcings compared to greenhouse gas forcing. Overwhelmed is mild.
  26. John Sully at 03:54 AM on 24 September, 2012
    And here we go again McIntyre is now claiming that the free market questions are also fraudulent. Steve, come back when you have something more than supposition. Comparing what Prof. Lewandowsky did and what Watts did is not appropriate since Watts let everyone know how they should answer in advance. This "pump-priming" automatically makes the Watts survey invalid.


    Oh really? Please point out exactly where this telling "everyone know how they should answer in advance" occured. I wrote the simple instructions for the survey. I wrote the guest post announcing it at WUWT. And I wrote the sentence or two that were additional to the original on the survey page.

    Please point out exactly where anyone was told how to vote? Please show exactly what "pre-priming" went on and where.

    I'd also be quite interested to see exactly how each of the 8 pro-AGW sites presented it. There you have 8 different sites, 8 different opportunities to "pre-prime" and 8 different blogs for the different groups to discuss how they were going to answer and scam the survey.

    Which we know occurred - both the discussions and the false responses.
  27. Sou at 13:03 PM on 24 September, 2012
    @301 John Sully - has A Scott or McI provided the data yet?

    Given that Watts' poll on his own website (just after the AScott survey IIRC) showed 99% of his readers reject climate science, this poll should reflect the same all things being equal - or be even more skewed toward science rejectors (since the avowed purpose of the poll was to discredit the Lewandowsky et al paper, proportionally even more 'skeptics' would be expected to respond in their enthusiasm).


    Nope - data will be provided when survey and analysis are done. Unlike the original, that won't take 2 years.

    And where the heck do you people get some of these silly and ridiculous claims?

    The reason for the survey was clearly identified. Lewandowsky wrote a paper whose purpose was to find links between skeptics, conspiracy and free market beliefs, and a resultant "motivated rejection of science."

    To find out what skeptics thought he went to pro-AGW sites - most fiercely anti-skeptic. (-snip-).

    (-snip-).

    (-snip-).

    The title "Moon Landing Hoax claim is made on the basis of 10 affirmative posts. Out of over 1300 total responses, 1,145 used in the study, a whopping 6 people strongly agreed and 4 more Agreed the Moon Landing was a hoax.

    (-snip-).

    (-snip-).

    The stated purpose, and my intent, were exactly the same ... to obtain a more robust, larger, quality set of survey responses from actual skeptics - which the authors failed to do. And then to transparently analyze that data to find out what the skeptics really believe.

    I'll repeat - the purpose was to obtain data, about skeptics, from skeptics, and in a quantity and quality that allowed legitimate, meaningful and useful statistical analysis to be done.
    Moderator Response: Multiple inflammatory and sloganeering snipped.
  28. A Scott @431 asks where the "pump priming occurred":

    Approx 20 minutes after posting the survey:
    "omg, what a leftist poll. Many questions lead to catch-22 answers. The questions refuse to acknowledge that environment can be valued in a free market system. The questions pit free market against environment, and involve Marxist social justice."
    http://wattsupwiththat.com/2012/09/08/replication-of-lewandowsky-survey/#comment-1074101

    16 minutes later:
    "I’ve completed the survey, but what is it supposed to prove? That we aren’t all nutters?

    Why not put up the “When did you stop beating your wife” survey?

    Can’t help feeling we’re playing so far in the crazy alarmist ballpark that we’ll get lost."
    http://wattsupwiththat.com/2012/09/08/replication-of-lewandowsky-survey/#comment-1074115

    And another 7 minutes later:

    "I added the following feedback…

    Bad phrasing:

    “An economic system based on free markets unrestrained by government interference automatically works best to meet human needs. *”

    No not always automatically. If phrased as “tends to” then would get strong agreement from me. As is I disagree.

    “4. The preservation of the free market system is more important than localized environmental concerns *”

    Define “local”. We’re trashing the Amazon or an empty field at the edge of a city where frogs live?

    “Smoking causes lung cancer *”

    It doesn’t cause lung cancer, but it does increase the risk. You can smoke and not get lung cancer; you can not smoke and get lung cancer.

    “Human CO2 emissions cause climate change *”

    In sufficient quantity CAN, that doesn’t mean always will."
    http://wattsupwiththat.com/2012/09/08/replication-of-lewandowsky-survey/#comment-1074117

    So, on the same page as the survey we have clear discussion indicating that the free market questions were such that strong agreement would be absurd. Similar discussion can be found in a variety of preceding blogs on Lewandowsky et al (in press). This alone is enough to so contaminate the results of your survey as to render them worthless.

    On top of that, people will tend (IMO) to assign equal partitions between different options offered to them. Given a choice between four options, for example, they will assign values out of 1 of 0.125, 0.375, 0.625, and 0.875, with 0 being complete disagreement and 1 being complete agreement. Given a choice of 5 values, the values the assigned weights are likely to shift towards 0.1, 0.3, 0.5, 0.7, and 0.9. That shift in values will mean people on torn between answer 3 (0.625) or 4 (0.875) with four available responses are more likely to answer 4 (0.7) than 5 (0.9) given a choice of five responses. McIntyre's assumption that the "strongly agree" responses are directly comparable between the two surveys is not warranted.
  29. A.Scott,

    Do you really not read Willard's blog? Before you came up with your bowdlerized version of the Lewandowski questionnaire Willard posted several posts of the tone "this can't be right", "we can't be this crazy" or whatever his misreading of the study was (IIRC, these were posted under the title of "The Daily Lew" among others). This was the pump priming. As far as the blogs which Prof. Lewandowski posted to: most of them had a pretty anodyne statement (I think that most of them are reproduced in a comment in another thread in this series). Hot Topic had a statement which could be taken as prejudicing the results (I certainly thought so).

    But I'll just let you continue providing data for the followup studies.
  30. #432 illustrates internal inconsistency, among other things.
  31. Sou at 13:03 PM on 24 September, 2012
    /joke - Has anyone started a campaign yet - to complain to their employer and funding body (eg about misusing IP, hiding the data and code), and to send McI/AScott FOI requests and emails and blog comments chanting "We want the data, release the code, show us every step of your workings, give us your emails!!!" /endjoke


    (-snip-). Nothing more - nothing less.

    As I see and understand it code is being posted essentially real time as they work on this. So others can see and critique their work

    Lewandowsky still has provided nothing, despite releasing this work publicly and promoting in the press as far back as July.

    Sure looks to me like code is being posted publicly as they go - with each step of their work . (-snip-).

    When the work on the re-created survey is to the point where Lewandowskys was - complete, released publicly and being promoted in the media as fact by the authors - (-snip-).

    There is a significant difference between reviewing the results of a "peer reviewed" "in press" work which has been released to the public and media and is being promoted by the author - mining for every sensationalized headline possible .... and your attempt to attack a work in process, and at its early stages no less.

    Regardless - as you point out - the difference is nite and day.

    The original authors still have not released the supplemental information, the code nor any of the methodology required to review their published work. Despite it being public, and its conclusions used as support for inflammatory headlines around the world.

    The review authors are releasing their work as it is completed - as their review is in process. That code has already been used by another - who is well qualified in the field - to complete his own statistical review.
    Moderator Response: Inflammatory and sloganeering snipped.
  32. #436 - talks about 'That code' (by which I guess he means the LOG12 data) 'has already been used by another - who is well qualified in the field...'.

    Worth pointing out that the person I assume AScott is referring to who's done a factor analysis of LOG12 data explicitly states in his blog article that he is not an expert on the underlying theory and philosophy of factor analysis.

    Nor did he do the analysis as described in LOG12 - he departed from their approach in a number of ways for his own reasons which he elucidated, as well as the fact that he didn't read/follow what LOG12 actually did.

    He does seem to know quite a bit more the practical application of EFA/SEM than McI, but arguably has not got the same level of expertise and knowledge as the specialists who authored LOG12.
  33. Tom Curtis at 11:11 AM on 26 September, 2012

    So Tom ... a few user comments are pre-priming?

    Were there user comments made during the survey at any of the 8 pro-AGW blogs?

    You know for a fact there were - and we know for a fact that those discussions included a number that specifically discussed faking the results - purposely attempting to manipulate the data.

    You, it seems, would ignore the same potential with the original survey. The original, according to the paper, collected data from Aug thru Oct 2010.

    I collected the majority of the data in 24-48 hours.

    We know there were discussions to purposely game the original. I have seen zero report of the same regarding the re-creation - and I'm sure with all the people looking to attack if there had been any we would have seen them well publicized by now.

    Last - you of all people - should know the silliness of claiming most, a majority, or even more than a tiny handful would engage in a vast conspiracy to game their answers - especially on a survey whose entire purpose was for their views to be accurately represented.

    Or that any such effort could not be identified in reviewing and validating the data. And there are many options available to validate - from the very large data set relative to the original, to the many sources of other comparative validation data available.
  34. Tom Curtis

    There is a large amount of writing on 4 vs. 5 (or 7, 9, and even 21) point Likert ratings. In almost everything I read there was a strong indication that what you want to seem to claim a benefit is anything but.

    A 4 point scale - as you note - forces a decision that often may not reflect the actual answers of the participant - thus decreasing the quality of the answers. Most literature indicates a 5 or 7 point scale is strongly preferred, as it increases the granularity and thus the resultant quality of the responses.

    I believe exactly that result was seen in the data - and specifically on the questions where, at least to me as a layman, it would be seen.

    And if you really have an issue with the difference in Likert scales, perhaps you should ask the authors. Their work on the key conspiracy questions - central to their most sensationalized claims - was directly based on and validated against Swami 2009.

    Which used a 9 point Likert scale. Not a 4 point one as used by the authors here.
  35. BTW - my comment as quoted by AScott #436 was written two days ago. Seems it had a partial effect FWIW :D
  36. "The previous poster, the person I was replying to, said this. Not Kahan. Not Kahan's mates. The previous poster."

    Then you should have quoted it - but didn't.

    And you repeated it without disavowing it. Hard to take your comment as anything other than something you agree with.

    And still is, even after you note someone else said it - you seem to think it forms part of your argument.
  37. "No, but it only takes a single counter-example to show that all you have is a correlation, not an explanatory theory."

    Don't be silly and inconsistent.

    All Kahan et al has is a correlation, and yet you have spent days implying it is explanatory, i.e. "more scientific literacy leads to less alarm", even though it has been explicitly pointed out to you more than once that they did not claim to have identified causality.

    And no-one I'm aware of has argued that all on its ownsome it is an explanatory hypothesis for every individual - except you, in order to attack that strawman.

    Your argument that a few individuals disprove the existence of a widespread influence is ludicrous. No-one is asserting in the first place that the influence in question is universal, let alone the only factor!
  38. I truly love folks who use big fancy words.

    Take bowdlerize for example, just what does that kinda cool, fancy word mean?
    Merriam Webster says:
    : to expurgate (as a book) by omitting or modifying parts considered vulgar
    : to modify by abridging, simplifying, or distorting in style or content
    Dictionary.com:
    :to expurgate (a written work) by removing or modifying passages considered vulgar or objectionable.
    FreeDictionary:
    : To remove material that is considered offensive or objectionable from (a book, for example).
    Wiktionary:
    : To remove or alter those parts of a text considered offensive, vulgar, or otherwise unseemly.
    See also: censor, expunge, expurge, redact

    Funny that ... Mr. Sully seems to be saying, since I used the exact words, all of them, of the original, that he agrees those words were vulgar, objectionable, offensive or unseemly.

    I, and the vast majority who've taken the survey, strongly agree with you!

    Unfortunately however, we are stuck with them in this instance.

    Wise man once say "when it doubt, words you know prevent looking the fool"

    PS - I understand that's not really what he was saying - but what he thought it meant makes him look more foolish yet, so I thought humor was better alternative ... :-)
  39. "Can you explain how the truth you claim is so obvious escapes the grokkage of Freeman Dyson, Alan Carlin, Stephen McIntyre and Richard Lindzen?"

    Well, there is this little thing called motivated reasoning - you might have heard of it?

    And then there's Going Emeritus.

    And then there's the fact that as far as I can see three of them are operating outside of their sphere of expertise. And in that light, there's this thing called the Dunning Kruger Effect - and very smart people are arguably more susceptible to it than less smart people.

    And then there are psychological factors, including motivations for attention seeking and contrarianism, and potentially financial factors. Lindzen, for example, says things to the non-scientific public that he must know would bring down outrage if he tried to publish them in the scientific literature - including IIRC rejecting the strong link between tobacco smoking and various nasty diseases. In the literature he has repeatedly tried over the last decade or more (and good on him - it's part of the scientific process) to show that the mainstream conclusions are inaccurate - but failed miserably.

    What do you think explains his public pronouncements clearly being at strong odds with the evidence - as he of all people must know, given (a) his acknowledged past stature as one of the most proficient atmospheric scientists, and (b) his inability to make a significant dent in the mainstream conclusions from the evidence?

    I'm sure there's more...all of which should be enough to get you thinking - but I predict it won't.
  40. Sou at 13:22 PM on 26 September, 2012
    BTW - my comment as quoted by AScott #436 was written two days ago. Seems it had a partial effect FWIW :D


    No - not in the slightest. For those of us watching and reading, that has been his practice all along. Perhaps you could persuade the authors to show the same ... for their work which has been complete and released to the public now for months.

    If their work is robust, can stand up to scrutiny, had they simply done this at the start they could have stopped this in its tracks.
  41. 1st reference, definition 2. You blew the Likert scale.
  42. "If not, you're in a conundrum."

    Saying it does not make it so.

    There will always be a few who buck even the strongest evidence, so insisting on the standard that an explanation of how people form beliefs has to explain every individual's formed beliefs - with only a single factor, no less - is ludicrous.

    There is a real Flat Earth Society, for example, and it has some apparently real Ph.D. members, and they apparently seriously argue that satellite photos are bogus or faked, or that the geometry of the earth is such that perspective makes it look round from certain positions when it's really fairly flat or even concave, and so on. Does that place "into a conundrum" either the strong consensus that evidence indicates Earth is an oblate spheroid, or any evidence-based explanation of the influence of small numbers of factors on how people accept or reject evidence when forming beliefs?

    Of course not.

    (And then one notes that the fact that the truth escapes some people doesn't have any bearing on whether it is true or not...)
  43. Sometimes you've got to wonder if people read what they themselves copy and paste (re #443) (let alone understand what they copy and bowdlerize):

    2: to modify by abridging, simplifying, or distorting in style or content
  44. John Sully at 13:47 PM on 26 September, 2012
    1st reference, definition 2. You blew the Likert scale.


    By golly you have me on a technicality ... you are right.

    I did remove the objectionable, offensive and unseemly 4 point Likert scale, replacing it with the 5 point scale that most literature indicates is far preferred, and that significantly improves the quality of responses and subsequently the data.

    I guess you got me. Silly me.

    :-)
  45. #443+ Wise man once say "when it doubt, words you know prevent looking the fool"

    Not sure who the wise man was but please, don't let him stop you. Your posts are very revealing.
  46. Rather amusing that A. Scott goes off on a riff about the use of the word "bowdlerized" and totally ignores the clarification of the concerns about pump-priming that were made in the same comment.
  47. For information only. To Bowdlerise stems from the efforts of Thomas Bowdler to make Shakespeare suitable for women and children by removing phrases, passages etc he considered unsuitable. Later it was discovered it was his sister Henrietta who did most of the censoring/
  48. I find many of the assumptions peculiar, for example:

    - that survey responses were faster on one blog than on another lot of blogs

    - that a single blogger's site is more representative than multiple bloggers' websites

    - that a five point scale is an improvement on a four point scale

    - that certain people know what the hell they are doing (or talking about)

    - that one earns more kudos from copying and pasting a survey than does a research team that explores the literature, designs the research study, designs the survey instrument, conducts the survey, analyses the results in the context of the broader literature, prepares a paper, gets it through peer review and accepted for publication in a leading international psychology journal

    - that it's perfectly okay to just use someone else's survey instrument (even if not specifically labelled proprietary)

    - that a layperson, despite never having previously used the stats and having no expertise in the research discipline (and who makes 'naive' errors and flawed assumptions), has more 'authority' to make declarations about a highly specialised area than the disciplinary specialists who use those techniques in their daily work in academia and the private sector

    - that a blogger's analysis done on a survey promoted on a blog where 99% of visitors reject climate science, can or will be used to test all aspects of this particular study (no-one is discussing findings other than conspiracy ideation, but there are others of interest)

    - for starters...
  49. @- A Scott
    "The stated purpose, and my intent, were exactly the same ... to obtain a more robust, larger, quality set of survey responses from actual skeptics - which the authors failed to do. And then to transparently analyze that data to find out what the skeptics really believe. "

    Do you have any serious expectation that your results will show anything different from the common finding that the ideological beliefs {and conspiracist ideation} of a person shape their tendency to accept of reject climate science ?
  50. Izen - I should have added your point to my list of peculiar assumptions. It's probably the most peculiar of all.

Prev  1  2  3  4  5  6  7  8  9  10  11  12  Next

Comments Policy

Post a Comment

You need to be logged in to post a comment. Login via the left margin or register a new account.