A few nights ago, I was talking with a family friend about autism. He'd seen the website and wanted to get my feedback about certain thoughts he had about the spectrum. All was going well and I was very happy to be having the conversation...until he brought up something he thought contributed to autism. When he showed me the article, which opens by saying in huge letters "There is a correlation between _____ and autism," I had immediately had to fight a strong negative reaction, which I realized was very much disproportionate to what he'd said and presented. Why did I have that reaction? There are a lot of reasons, which I'll lay out and then elaborate on further.
Thinking about the situation, I realized there were actually two dimensions explaining my reaction, which, accordingly, will be discussed in two posts. The first part - this post - will center on my skepticism of what many sources have to say about autism, especially its cause. The second part will discuss the broader social reasons why I don't always fully agree with the discussions of the cause of autism.
In this post, we'll be focusing on my skepticism with regards to the one source my family friend presented, because to be honest, its flaws and failures and those of many other such articles, books, etc claiming to have discovered the cause of autism. Together we'll go through the important questions and criteria that many published sources claiming proposed causes of autism fail, so that when you're reading on your own, you can through these questions in your head before you even start to entertain the idea that the claim might be true. There are other questions and criteria which you should also use, but this should get a good foundation set for you.
Since I'm attempting to show underlying principles, I don't want people to get side-tracked with the proposed cause presented by the book, so we'll just use ____ to refer to it. I would advise against attempts to determine the proposed cause or book under discussion; I tested this with multiple people and their guesses were consistently wrong (it's a very obscure proposed cause).
The Questions:
1. What is the source?
2. What are the source's sources?
3. What is the claim?
4. How did the author come up with that claim?
1. What is the source?
In this case, the source was a book. We tend to think of books as being reliable and reputable, but they don't have to meet the same rigorous requirements as a journal article. For a journal article, when you're researching you can and should search for those articles which are published, in well respected journals, and have their sources and methodology available for all to see. You may not understand the methodology, but the mere fact that it's posted in the paper for other scientists to critique, replicate, etc is certainly much better evidence towards its credibility than is the fact that it's got a hardcover binding.
Another reason that books aren't as trustworthy as journal articles is that the process of peer review regulates the information that gets published in journals by making it undergo the close scrutiny of experts in whatever field the article is dealing with before it is published. The process isn't perfect, but it's a lot better than just trusting a publisher to do it instead. A publisher of books for mass consumption is looking for a book that will make a splash and lots of sales, which, one might argue, would lead books with sensationalist and attention-getting conclusions to get published more frequently. While sensationalist and accurate aren't necessarily mutually exclusive, they certainly don't make for good bedfellows. The process of peer-review, while it can still be said to look for articles with attention-getting conclusions, has a group of experts check those conclusions for scientific accuracy before it's published. The same cannot be said for a book.
2. What are the source's sources?
At this point, someone who believed in this book might say "Ah-ha! The book cites studies, so your argument fails!" At which point I would have to reply, it's funny you should mention that because there's yet another problem with the book - its own sources, which are case studies (to call them such is generous, but I'm in a generous mood) and flawed studies which never even made it through the peer review process.
The case studies cited by the book are really just some claims by authoritative sounding people. A pediatrician is quoted saying that she thinks that _____ is related to problems in people who aren't on the spectrum, and then _____ is therefore implied to cause problems in people with autism. It also consistently applies observations of problems that doctors, etc have seen in some animal subjects to people on the autism spectrum...despite there being nothing particularly autistic or simulating autism about the animals. In short, they're not good sources.
As for the flawed study, investigation of the wild new claim the study made quickly revealed many serious errors in the study. I can't tell you one without revealing what the subject is, but the other error was that the subjects (animals) involved were extremely mistreated, which would mess up the results of any study. The study was also replicated by another, more reputable, group which found there to be no significant relationship between the two items under investigation. The authors either (a) didn't read the study/methodology, (b) didn't understand it, or (c) read it, knew it was bad, and printed it anyways knowing that readers would be too trusting to look it up themselves. Unfortunately, (c) is all too true. When confronted with a footnote, very few people actually investigate the source. Don't let that person be you - before you believe a source's claims, always investigate its own sources...you never know what you'll find.
3. What is the claim?
Another problem with the claimed cause was the fact that, from the very first page, it seemed to encourage readers to mix up correlation and causation. As I said, on the very first page of the book it said opens by printing in huge letters "There is a correlation between _____ and autism." Ignoring the fact that they didn't even say what type of correlation (Negative or positive? Strong or weak? You'd never know from this splashy claim...and perhaps the author intends it that way) from the very beginning it encourages you to think that is a very strong evidence towards its claim. What it doesn't elaborate on is that correlation doesn't equal causation.
I showed this graph before, but we'll go through it again:
Thinking about the situation, I realized there were actually two dimensions explaining my reaction, which, accordingly, will be discussed in two posts. The first part - this post - will center on my skepticism of what many sources have to say about autism, especially its cause. The second part will discuss the broader social reasons why I don't always fully agree with the discussions of the cause of autism.
In this post, we'll be focusing on my skepticism with regards to the one source my family friend presented, because to be honest, its flaws and failures and those of many other such articles, books, etc claiming to have discovered the cause of autism. Together we'll go through the important questions and criteria that many published sources claiming proposed causes of autism fail, so that when you're reading on your own, you can through these questions in your head before you even start to entertain the idea that the claim might be true. There are other questions and criteria which you should also use, but this should get a good foundation set for you.
Since I'm attempting to show underlying principles, I don't want people to get side-tracked with the proposed cause presented by the book, so we'll just use ____ to refer to it. I would advise against attempts to determine the proposed cause or book under discussion; I tested this with multiple people and their guesses were consistently wrong (it's a very obscure proposed cause).
The Questions:
1. What is the source?
2. What are the source's sources?
3. What is the claim?
4. How did the author come up with that claim?
1. What is the source?
In this case, the source was a book. We tend to think of books as being reliable and reputable, but they don't have to meet the same rigorous requirements as a journal article. For a journal article, when you're researching you can and should search for those articles which are published, in well respected journals, and have their sources and methodology available for all to see. You may not understand the methodology, but the mere fact that it's posted in the paper for other scientists to critique, replicate, etc is certainly much better evidence towards its credibility than is the fact that it's got a hardcover binding.
Another reason that books aren't as trustworthy as journal articles is that the process of peer review regulates the information that gets published in journals by making it undergo the close scrutiny of experts in whatever field the article is dealing with before it is published. The process isn't perfect, but it's a lot better than just trusting a publisher to do it instead. A publisher of books for mass consumption is looking for a book that will make a splash and lots of sales, which, one might argue, would lead books with sensationalist and attention-getting conclusions to get published more frequently. While sensationalist and accurate aren't necessarily mutually exclusive, they certainly don't make for good bedfellows. The process of peer-review, while it can still be said to look for articles with attention-getting conclusions, has a group of experts check those conclusions for scientific accuracy before it's published. The same cannot be said for a book.
2. What are the source's sources?
At this point, someone who believed in this book might say "Ah-ha! The book cites studies, so your argument fails!" At which point I would have to reply, it's funny you should mention that because there's yet another problem with the book - its own sources, which are case studies (to call them such is generous, but I'm in a generous mood) and flawed studies which never even made it through the peer review process.
The case studies cited by the book are really just some claims by authoritative sounding people. A pediatrician is quoted saying that she thinks that _____ is related to problems in people who aren't on the spectrum, and then _____ is therefore implied to cause problems in people with autism. It also consistently applies observations of problems that doctors, etc have seen in some animal subjects to people on the autism spectrum...despite there being nothing particularly autistic or simulating autism about the animals. In short, they're not good sources.
As for the flawed study, investigation of the wild new claim the study made quickly revealed many serious errors in the study. I can't tell you one without revealing what the subject is, but the other error was that the subjects (animals) involved were extremely mistreated, which would mess up the results of any study. The study was also replicated by another, more reputable, group which found there to be no significant relationship between the two items under investigation. The authors either (a) didn't read the study/methodology, (b) didn't understand it, or (c) read it, knew it was bad, and printed it anyways knowing that readers would be too trusting to look it up themselves. Unfortunately, (c) is all too true. When confronted with a footnote, very few people actually investigate the source. Don't let that person be you - before you believe a source's claims, always investigate its own sources...you never know what you'll find.
3. What is the claim?
Another problem with the claimed cause was the fact that, from the very first page, it seemed to encourage readers to mix up correlation and causation. As I said, on the very first page of the book it said opens by printing in huge letters "There is a correlation between _____ and autism." Ignoring the fact that they didn't even say what type of correlation (Negative or positive? Strong or weak? You'd never know from this splashy claim...and perhaps the author intends it that way) from the very beginning it encourages you to think that is a very strong evidence towards its claim. What it doesn't elaborate on is that correlation doesn't equal causation.
I showed this graph before, but we'll go through it again:
Even for those of you with a statistics background, this graph might initially pass the test, since r=.9971 would lead you to believe that the correlation is very strong (although I'm not sure where this number came from and I don't think the graph actually reflects the supposed correlation). But correlation is not the same as causation, so even if this claim is true, as you can probably guess, that doesn't mean that organic food causes autism (or autism causes organic food).
Also, remember how I said to always check the footnotes? Here's another example of how important that is. When you look up the studies mentioned in the bottom, they don't provide the data given in the graph. The data's made up - the footnotes are just there to make it look official. (Although in real life, organic food consumption over time and autism diagnoses likely do have a positive correlation...but the reason is simply because both have been separately increasing over time.)
It turns out, of course, that this isn't a real claim at all - just something someone made while they were messing around to show the perils of confusing correlation and causation. After reading this, I hope you'll be cautious the next time someone tells you that two things are correlated - yes, that can mean there's a causal relationship, but it can just as easily mean nothing at all.
So, returning to the claim under discussion, the fact that they appear to be presenting "There is a correlation between _____ and autism" as strong, even damning, evidence pointing to a causal relationship between autism and ____ is something you should be incredibly skeptical of - so skeptical, in fact, that it should cause you to regard the entire work with an increased level of scrutiny.
Also, remember how I said to always check the footnotes? Here's another example of how important that is. When you look up the studies mentioned in the bottom, they don't provide the data given in the graph. The data's made up - the footnotes are just there to make it look official. (Although in real life, organic food consumption over time and autism diagnoses likely do have a positive correlation...but the reason is simply because both have been separately increasing over time.)
It turns out, of course, that this isn't a real claim at all - just something someone made while they were messing around to show the perils of confusing correlation and causation. After reading this, I hope you'll be cautious the next time someone tells you that two things are correlated - yes, that can mean there's a causal relationship, but it can just as easily mean nothing at all.
So, returning to the claim under discussion, the fact that they appear to be presenting "There is a correlation between _____ and autism" as strong, even damning, evidence pointing to a causal relationship between autism and ____ is something you should be incredibly skeptical of - so skeptical, in fact, that it should cause you to regard the entire work with an increased level of scrutiny.
4. How did it come up with that claim?
Not only did it cite bad studies and case studies, and rely on correlation to show causation, but the book's logic for its finale was also very flawed. It says "people on the autism spectrum have ____ problem" and "animals exposed to _____ have the same problem" and jumps to "_____ must cause these symptoms experienced by people on the spectrum!" That proves nothing. To give you an example using that same logic, I can say "parents of newborns are tired" and "animals that exercise a lot are tired" - but if I claimed that "exercise is what causes parents of newborns to be tired" you'd just roll your eyes at me because, as we all know, parents of newborns are probably tired because they don't get much sleep. Likewise, you should roll your eyes at similar claims.
Conclusion
My goal in writing this was not to mock the book nor to mock anyone who has ever believed a source that failed many of these criteria. I've fallen victim to it myself in the past, and have learned my lesson and am happy to pass it on. Especially when a relative (or yourself) is first diagnosed, it's incredibly difficult to sort through all of the information hurtling your way and figure out what is just noise and what is signal. One father of a newly diagnosed child likened it to "trying to drink out of a fire hose." It is my hope that this guide can help you determine which sources are reliable and which are not. Good luck!
And stay tuned for the second half of this blog post, my concerns about the discussion of the cause of autism in society.
Not only did it cite bad studies and case studies, and rely on correlation to show causation, but the book's logic for its finale was also very flawed. It says "people on the autism spectrum have ____ problem" and "animals exposed to _____ have the same problem" and jumps to "_____ must cause these symptoms experienced by people on the spectrum!" That proves nothing. To give you an example using that same logic, I can say "parents of newborns are tired" and "animals that exercise a lot are tired" - but if I claimed that "exercise is what causes parents of newborns to be tired" you'd just roll your eyes at me because, as we all know, parents of newborns are probably tired because they don't get much sleep. Likewise, you should roll your eyes at similar claims.
Conclusion
My goal in writing this was not to mock the book nor to mock anyone who has ever believed a source that failed many of these criteria. I've fallen victim to it myself in the past, and have learned my lesson and am happy to pass it on. Especially when a relative (or yourself) is first diagnosed, it's incredibly difficult to sort through all of the information hurtling your way and figure out what is just noise and what is signal. One father of a newly diagnosed child likened it to "trying to drink out of a fire hose." It is my hope that this guide can help you determine which sources are reliable and which are not. Good luck!
And stay tuned for the second half of this blog post, my concerns about the discussion of the cause of autism in society.
RSS Feed