How to Assess Claims &
Draw Your Own Conclusions
The ability to evaluate a scientific claim is one of the most important abilities that I would argue anyone can cultivate, but it is particularly important among those who care about people with autism. You will hear many claims about autism, particularly about its cause and purported treatments. Many of those claims will be false, but it's up to you to be able to evaluate them and recognize that fact.
In this article, I'm going to teach you how to do your own research into claims and how to evaluate the claims that you find. At the end I'm going to give you an abstract about common autism-related claims and ask you to evaluate it on your own for practice. (There will be a key at the bottom.)
This is not going to be a "fun", inspirational, or enjoyable article, but I strongly feel it should be required reading for every parent of an autistic child, so particularly if you fall into that category, please stick with it and read the article all the way through.
In this article, I'm going to teach you how to do your own research into claims and how to evaluate the claims that you find. At the end I'm going to give you an abstract about common autism-related claims and ask you to evaluate it on your own for practice. (There will be a key at the bottom.)
This is not going to be a "fun", inspirational, or enjoyable article, but I strongly feel it should be required reading for every parent of an autistic child, so particularly if you fall into that category, please stick with it and read the article all the way through.
Why You Should Do Your Own Scientific Research
This is one of those rare cases in which Google - or at least general Google - is not your friend. For example, let's say you're looking to figure out if eliminating _______ helps with autism. You'd probably Google "_____ and autism". Not including the blurb about Google Scholar at the top, which most people skip, it's unlikely that a single scientific study would emerge on the first results page. Instead, you'd likely be left with loads of blog articles, which are glorified opinions that generally don't even come from experts, and maybe a few news articles if you're lucky.
That's not good.
Instead, what you should be doing is going to a scholarly article search engine so that you can ensure you're looking at scientific studies instead of articles anyone could write.
Why is reading scientific studies superior to just reading articles about them? Well, if you dig hard enough and don't particularly care about the scientific accuracy of a study, or don't know the difference between correlation and causation, you can find studies that say practically anything. I can find studies that tell you cigarettes don't cause lung cancer, which is a claim that some hard core cigarette smokers cling to but hopefully all you readers know is patently false, so that should tell you something. Most studies are well made and their results are accurate, but there are an occasional few which are not (I'll teach you how to sort those out from the others later).
If you go somewhere other than a scholarly article search engine, you have no way of knowing if the website you're viewing is picking and choosing studies - regardless of accuracy - to reflect the website's point of view. Sometimes the fact that is occuring is fairly obvious. For instance, if you go to a website called "_____causesautism.com," are they going to have any studies on there that don't show _____ causes autism? It's unlikely to say the least. But at other points it's more insidious. For instance, maybe you go to a website called "Institute for the Study of Autism" (which I'm hoping doesn't really exist, because I'm just using this as an example). Sounds pretty official and unbiased, doesn't it? But that organization is just as capable believing _____ causes autism or _____ is a good treatment for autism and of filtering the information they're giving you to better support those beliefs, even if they don't say quite as openly what those beliefs are.
For that reason, it is imperative that you be able to do your own research and interpret claims on your own using scholarly search engines.
That's not good.
Instead, what you should be doing is going to a scholarly article search engine so that you can ensure you're looking at scientific studies instead of articles anyone could write.
Why is reading scientific studies superior to just reading articles about them? Well, if you dig hard enough and don't particularly care about the scientific accuracy of a study, or don't know the difference between correlation and causation, you can find studies that say practically anything. I can find studies that tell you cigarettes don't cause lung cancer, which is a claim that some hard core cigarette smokers cling to but hopefully all you readers know is patently false, so that should tell you something. Most studies are well made and their results are accurate, but there are an occasional few which are not (I'll teach you how to sort those out from the others later).
If you go somewhere other than a scholarly article search engine, you have no way of knowing if the website you're viewing is picking and choosing studies - regardless of accuracy - to reflect the website's point of view. Sometimes the fact that is occuring is fairly obvious. For instance, if you go to a website called "_____causesautism.com," are they going to have any studies on there that don't show _____ causes autism? It's unlikely to say the least. But at other points it's more insidious. For instance, maybe you go to a website called "Institute for the Study of Autism" (which I'm hoping doesn't really exist, because I'm just using this as an example). Sounds pretty official and unbiased, doesn't it? But that organization is just as capable believing _____ causes autism or _____ is a good treatment for autism and of filtering the information they're giving you to better support those beliefs, even if they don't say quite as openly what those beliefs are.
For that reason, it is imperative that you be able to do your own research and interpret claims on your own using scholarly search engines.
How To Find Quality Scientific Studies
Not all studies are made alike, and even if a study lacks any flaws whatsoever, the results of that study may not serve as strong evidence one way or another. Here we'll learn about the different types of studies, how to find them, and how you should weight their evidence.
Above is the pyramid of evidence based medicine, which is a great graphic to show you how much weight you should give to the evidence presented by different types of studies. The types of studies at the very top are those you should give the most weight - the types at the bottom should be given the least. Below we'll go through the categories of studies (filtered information, unfiltered information, and background information/expert opinion), and their subcategories, plus what scholarly search engines will let you search for what quality information. We'll start with the top of the pyramid, with filtered information, and work our way down.
Filtered Resources
Filtered resources appraise the quality of studies and often make recommendations for practice. All of these resources fall within the top half of the evidence based medicine pyramid.
Systematic Reviews
Authors of a systematic review ask a specific clinical question, perform a comprehensive literature search, eliminate the poorly done studies and attempt to make practice recommendations based on the well-done studies. Basically, they do the work that non-scientists (as you presumably are) can't do and make sure that you're looking at high quality material. For that reason, even more than for most cases, I highly recommend searching for systematic reviews (or at least other filtered information) when you can, because the science has already been interpreted for you and you won't be sorting through trying to figure out what studies are well made and which are bad.
Scholarly search engines you can use to find systematic reviews:
The Cochrane Database of Systematic Reviews
Cochrane Reviews are systematic reviews of completed by teams of experts combing through primary research in human health care who evaluate literature and present summaries of the findings of the best studies. They are internationally recognised as the highest standard in evidence-based health care and investigate the effects of interventions for prevention, treatment and rehabilitation.
Each systematic review addresses a clearly formulated question; for example: Can antibiotics help in alleviating the symptoms of a sore throat? All the existing primary research on a topic that meets certain criteria is searched for and collated, and then assessed using stringent guidelines, to establish whether or not there is conclusive evidence about a specific treatment. The reviews are updated regularly, ensuring that treatment decisions can be based on the most up-to-date and reliable evidence.
The nice thing about systematic reviews is that if there is not enough high quality evidence yet to make an assessment about a treatment one way or another, they will tell you that. While blogs and articles will have you convinced that they know the true answer, systematic scientific reviews are not afraid to say when we don't yet have enough information. That is not reason, mind you, to take that as permission to make your own decision as to what the truth is. Science is generally pointing one way or another (and they'll generally show what direction that is in the reviews). All it means is that we don't have sufficient evidence to know for sure yet.
The only problem with systematic reviews is that there aren't so comparatively many of them and they're only done in areas that have been well studied. So if you want to know whether something very recent causes autism or have a more obscure question, while I'd still suggest checking the systematic reviews first, they're less likely to have the answer. That said, if you want to know the effectiveness of some of the more common traditional treatments, such as risperidone and SSRIs, or alternative treatments, such as vitamin B6 and magnesium or acupuncture or a gluten free diet, this is the place to turn.
Check out the Cochrane Database here. I recommend just searching autism and then reading up on the reviews they have written, which are written in clear, concise, easy to understand language. Don't let that mislead you - a lot of work and a lot of reading and interpreting many, many studies went into this. However, for better interpretation by the general public it's not written in science-ese, a fact for which I'm sure we're all grateful.
Critically-Appraised Topics
Authors of critically-appraised topics evaluate and synthesize multiple research studies. While there isn't a review of all the literature necessarily, so it's not quite as good as a systematic review, it's still using multiple studies.
National Guideline Clearinghouse
A comprehensive database of evidence-based clinical practice guidelines and related documents produced by the Agency for Health Care Research and Quality, in partnership with the American Medical Association and the American Association of Health Plans. Updated weekly. Note: Guideline evidence varies from expert opinion to high levels of evidence.
Critically Appraised Individual Articles
Authors of critically-appraised individual articles evaluate and synopsize individual research studies.
Bandolier
Bandolier is an independent journal about evidence-based healthcare published in the UK. It includes "information about evidence of effectiveness (or lack of it), and put[s] the results forward as simple bullet points of those things that worked and those that did not: a bandolier with bullets. Information comes from systematic reviews, meta-analyses, randomised trials, and from high quality observational studies."
Scholarly search engines you can use to find systematic reviews:
The Cochrane Database of Systematic Reviews
Cochrane Reviews are systematic reviews of completed by teams of experts combing through primary research in human health care who evaluate literature and present summaries of the findings of the best studies. They are internationally recognised as the highest standard in evidence-based health care and investigate the effects of interventions for prevention, treatment and rehabilitation.
Each systematic review addresses a clearly formulated question; for example: Can antibiotics help in alleviating the symptoms of a sore throat? All the existing primary research on a topic that meets certain criteria is searched for and collated, and then assessed using stringent guidelines, to establish whether or not there is conclusive evidence about a specific treatment. The reviews are updated regularly, ensuring that treatment decisions can be based on the most up-to-date and reliable evidence.
The nice thing about systematic reviews is that if there is not enough high quality evidence yet to make an assessment about a treatment one way or another, they will tell you that. While blogs and articles will have you convinced that they know the true answer, systematic scientific reviews are not afraid to say when we don't yet have enough information. That is not reason, mind you, to take that as permission to make your own decision as to what the truth is. Science is generally pointing one way or another (and they'll generally show what direction that is in the reviews). All it means is that we don't have sufficient evidence to know for sure yet.
The only problem with systematic reviews is that there aren't so comparatively many of them and they're only done in areas that have been well studied. So if you want to know whether something very recent causes autism or have a more obscure question, while I'd still suggest checking the systematic reviews first, they're less likely to have the answer. That said, if you want to know the effectiveness of some of the more common traditional treatments, such as risperidone and SSRIs, or alternative treatments, such as vitamin B6 and magnesium or acupuncture or a gluten free diet, this is the place to turn.
Check out the Cochrane Database here. I recommend just searching autism and then reading up on the reviews they have written, which are written in clear, concise, easy to understand language. Don't let that mislead you - a lot of work and a lot of reading and interpreting many, many studies went into this. However, for better interpretation by the general public it's not written in science-ese, a fact for which I'm sure we're all grateful.
Critically-Appraised Topics
Authors of critically-appraised topics evaluate and synthesize multiple research studies. While there isn't a review of all the literature necessarily, so it's not quite as good as a systematic review, it's still using multiple studies.
National Guideline Clearinghouse
A comprehensive database of evidence-based clinical practice guidelines and related documents produced by the Agency for Health Care Research and Quality, in partnership with the American Medical Association and the American Association of Health Plans. Updated weekly. Note: Guideline evidence varies from expert opinion to high levels of evidence.
Critically Appraised Individual Articles
Authors of critically-appraised individual articles evaluate and synopsize individual research studies.
Bandolier
Bandolier is an independent journal about evidence-based healthcare published in the UK. It includes "information about evidence of effectiveness (or lack of it), and put[s] the results forward as simple bullet points of those things that worked and those that did not: a bandolier with bullets. Information comes from systematic reviews, meta-analyses, randomised trials, and from high quality observational studies."
Unfiltered Resources
Evidence is not always available via filtered resources, particularly for less common claims. That's where unfiltered resources come in, which will give you links to randomized controlled studies, cohort studies and case studies. Randomized controlled studies are the best of the three, as they can actually point to causation. Cohort studies and case studies can only point to correlation, which is not nearly as strong of evidence since it doesn't show the direction of an observed relationship (is it autism that causes ____ or ____ that causes autism?) nor does it control for confounding variables (so for instance if you have a study that shows X is correlated with autism, you have no way of knowing if Y couldn't be what's actually causing autism, and X is just correlated with that).
We'll cover the pitfalls more in depth later. Suffice it to say for now, if a study is observational in nature (meaning you're not actually controlling variables, simply watching what subjects are like), it's a cohort or case study and therefore weak evidence. Here's an example of what a cohort study is like and an example of what a randomized control trial study is like (the fact that this study is double blind and placebo controlled are extra qualities of a good study that we'll go over later) so you can better identify them. So while you're looking through the journal articles these resources will link you to, look for randomized control studies (whose
PubMed
PubMed comprises more than 23 million citations for biomedical literature from MEDLINE, life science journals, and online books.
Google Scholar
Provides a search of scholarly literature across many disciplines and sources, including theses, books, abstracts and articles.
We'll cover the pitfalls more in depth later. Suffice it to say for now, if a study is observational in nature (meaning you're not actually controlling variables, simply watching what subjects are like), it's a cohort or case study and therefore weak evidence. Here's an example of what a cohort study is like and an example of what a randomized control trial study is like (the fact that this study is double blind and placebo controlled are extra qualities of a good study that we'll go over later) so you can better identify them. So while you're looking through the journal articles these resources will link you to, look for randomized control studies (whose
PubMed
PubMed comprises more than 23 million citations for biomedical literature from MEDLINE, life science journals, and online books.
Google Scholar
Provides a search of scholarly literature across many disciplines and sources, including theses, books, abstracts and articles.
Background Information/Expert Opinion
This is the absolute least strong evidence that you can find. Basically what this constitutes is just articles you'll find online from physicians and scientists expressing their thoughts on a topic - it's anything other than a study. It still has its place and can be useful, but it is incredibly weak evidence and really shouldn't sway your opinions or beliefs much one way or another.
eMedicine
Physician authors and editors contribute to the eMedicine Clinical Knowledge Base, which contains articles on 7,000 diseases and disorders. The evidence-based content provides the latest practice guidelines in 62 medical specialties. eMedicine's professional content undergoes multiple levels of physician peer review. Free resource, but you must first make an account to use.
Please note that expert opinion is at the VERY bottom of the evidence based pyramid. That's for a reason, since expert advice varies from expert to expert and is subject to a great many human biases and flaws. If expert opinion is considered to be such weak evidence, I'm sure you can imagine where the advice of a layperson who happens to be related to someone with autism (that's right, I'm including myself here) on scientific matters would fall. It's so weak, it's not even on the chart.
You should not take scientific advice solely on the basis of this or any website (that is not .gov or .edu), or from myself or any layperson. The whole point of this page is teaching you to do your own research because others' opinions and advice (and even trying things on your own, like doing _____ with your autistic child to see what will happen) is incredibly weak evidence which you should not take at face value. You'll note, I stay out of passing judgment on treatments, cures, etc in this website (unless I view the issue as absolutely scientifically cut and dried) because I am just plain not qualified to discuss it. Later, I may make a page with important studies to get you started, but I do not seek to offer my own thoughts because they're simply not terribly valuable. And neither are anyone else's. You'll need to sort through the research on your own and draw your own conclusions from it.
eMedicine
Physician authors and editors contribute to the eMedicine Clinical Knowledge Base, which contains articles on 7,000 diseases and disorders. The evidence-based content provides the latest practice guidelines in 62 medical specialties. eMedicine's professional content undergoes multiple levels of physician peer review. Free resource, but you must first make an account to use.
Please note that expert opinion is at the VERY bottom of the evidence based pyramid. That's for a reason, since expert advice varies from expert to expert and is subject to a great many human biases and flaws. If expert opinion is considered to be such weak evidence, I'm sure you can imagine where the advice of a layperson who happens to be related to someone with autism (that's right, I'm including myself here) on scientific matters would fall. It's so weak, it's not even on the chart.
You should not take scientific advice solely on the basis of this or any website (that is not .gov or .edu), or from myself or any layperson. The whole point of this page is teaching you to do your own research because others' opinions and advice (and even trying things on your own, like doing _____ with your autistic child to see what will happen) is incredibly weak evidence which you should not take at face value. You'll note, I stay out of passing judgment on treatments, cures, etc in this website (unless I view the issue as absolutely scientifically cut and dried) because I am just plain not qualified to discuss it. Later, I may make a page with important studies to get you started, but I do not seek to offer my own thoughts because they're simply not terribly valuable. And neither are anyone else's. You'll need to sort through the research on your own and draw your own conclusions from it.
EBM Pyramid and EBM Page Generator. (c) Copyright 2006-2011. Trustees of Dartmouth College and Yale University. All Rights Reserved. Produced by Jan Glover, David Izzo, Karen Odato and Lei Wang.
How Can You Tell if a Study is Bad?
As I said previously, not all studies are made alike, and you can find some studies that have patently false claims (the example I cited was studies that show you smoking does not cause cancer). How, then, can you tell those studies apart from others?
Well, using the filtered studies will take care of a lot of that problem, but let's say you can't find what you're looking for on the filtered search sites and have to resort to unfiltered information. What then? Well, there's one big red flag I use to help me determine if a study might be inaccurate. The study's findings are different from, or even opposite to, the findings of many other studies.
So to use smoking as an example again, if you search cigarette smoking and lung cancer on Google Scholar, you'll likely find many, many studies that all say smoking causes lung cancer. If you search long and hard enough, you may be able to find a study that has more mixed results or says that smoking does not cause lung cancer (such a study would probably be from many decades ago, but there probably is one out there). The fact that this one study flies in the face of mountains of evidence, though, should be a giant red flag for you, warning you that the study is likely to either have a critical flaw or be purposefully fraudulent.
If there's not enough research out there for you to be able to detect a trend one way or another, but you still are concerned that a study might be fraudulent, that is when I might turn to Google and see if there are articles (not blog posts!) evaluating the study.
Well, using the filtered studies will take care of a lot of that problem, but let's say you can't find what you're looking for on the filtered search sites and have to resort to unfiltered information. What then? Well, there's one big red flag I use to help me determine if a study might be inaccurate. The study's findings are different from, or even opposite to, the findings of many other studies.
So to use smoking as an example again, if you search cigarette smoking and lung cancer on Google Scholar, you'll likely find many, many studies that all say smoking causes lung cancer. If you search long and hard enough, you may be able to find a study that has more mixed results or says that smoking does not cause lung cancer (such a study would probably be from many decades ago, but there probably is one out there). The fact that this one study flies in the face of mountains of evidence, though, should be a giant red flag for you, warning you that the study is likely to either have a critical flaw or be purposefully fraudulent.
If there's not enough research out there for you to be able to detect a trend one way or another, but you still are concerned that a study might be fraudulent, that is when I might turn to Google and see if there are articles (not blog posts!) evaluating the study.
How Do You Interpret a Study's Findings?
Reading a full scientific study can be rather intimidating for a layperson. For that reason, along with the fact that in many databases the abstract is all that's accessible to read, I'm only going to walk you through how to interpret the abstract. The abstract of a study is essentially a short summary of the essentials of the study - who was studied, why they were studied, how the study was conducted, and what the results were.
Below I've inserted the abstract of a real study related to autism. I edited the abstract to have a format more similar to most abstracts you'll encounter. This study is specifically related to one of the comorbid problems with autism, anxiety. We're going to analyze it in depth in a moment, but for now as you read it, try to figure out whether it's a randomized controlled trial or an observational study (cohort study or case study). Remember, randomized controlled trials can point to causation, while observational studies can only point to correlation (say that two things are related).
Abstract
"The objective was to examine the efficacy of a modular cognitive-behavioral therapy (CBT) protocol relative to treatment as usual (TAU) among children with high-functioning autism spectrum disorders (ASD) and clinically significant anxiety. A total of 45 children (7-11 years of age) with high-functioning ASD and clinically significant anxiety were randomized to receive 16 sessions of weekly CBT or TAU for an equivalent duration. After screening, assessments were conducted at baseline, post-treatment, and 3-month follow-up. Raters were blind to treatment condition. Youth receiving CBT showed substantial improvement relative to TAU on primary anxiety outcomes. Of 24 children randomized to the CBT arm, 18 (75%) were treatment responders, versus only 3 of 21 children (14%) in the TAU arm. Gains were generally maintained at 3-month follow-up for CBT responders. Relative to usual care, CBT adapted for anxious youth with high-functioning ASD demonstrates large effects in reducing anxiety symptoms. This study contributes to the growing literature supporting adapted CBT approaches for treating anxiety in youth with ASD."
Storch, E., Arnold, E., Lewin, A., Nadeau, J., & Jones, A. (2013). The Effect of Cognitive-Behavioral Therapy Versus Treatment as Usual for Anxiety in Children With Autism Spectrum Disorders: A Randomized, Controlled Trial.Journal of the American Academy of Child & Adolescent Psychiatry, 132-142.
Retrieved December 6, 2014, from http://www.ncbi.nlm.nih.gov/pubmed/23357440
Below I've inserted the abstract of a real study related to autism. I edited the abstract to have a format more similar to most abstracts you'll encounter. This study is specifically related to one of the comorbid problems with autism, anxiety. We're going to analyze it in depth in a moment, but for now as you read it, try to figure out whether it's a randomized controlled trial or an observational study (cohort study or case study). Remember, randomized controlled trials can point to causation, while observational studies can only point to correlation (say that two things are related).
Abstract
"The objective was to examine the efficacy of a modular cognitive-behavioral therapy (CBT) protocol relative to treatment as usual (TAU) among children with high-functioning autism spectrum disorders (ASD) and clinically significant anxiety. A total of 45 children (7-11 years of age) with high-functioning ASD and clinically significant anxiety were randomized to receive 16 sessions of weekly CBT or TAU for an equivalent duration. After screening, assessments were conducted at baseline, post-treatment, and 3-month follow-up. Raters were blind to treatment condition. Youth receiving CBT showed substantial improvement relative to TAU on primary anxiety outcomes. Of 24 children randomized to the CBT arm, 18 (75%) were treatment responders, versus only 3 of 21 children (14%) in the TAU arm. Gains were generally maintained at 3-month follow-up for CBT responders. Relative to usual care, CBT adapted for anxious youth with high-functioning ASD demonstrates large effects in reducing anxiety symptoms. This study contributes to the growing literature supporting adapted CBT approaches for treating anxiety in youth with ASD."
Storch, E., Arnold, E., Lewin, A., Nadeau, J., & Jones, A. (2013). The Effect of Cognitive-Behavioral Therapy Versus Treatment as Usual for Anxiety in Children With Autism Spectrum Disorders: A Randomized, Controlled Trial.Journal of the American Academy of Child & Adolescent Psychiatry, 132-142.
Retrieved December 6, 2014, from http://www.ncbi.nlm.nih.gov/pubmed/23357440
What type of study was it?
Did you think it was an observational study or a randomized controlled trial? If you picked randomized controlled trial, you're right! If you picked observational study, read up more on what a randomized controlled trial (RCT) consists of.
Now try to answer the following questions based on the abstract as best you can on your own. Answers (and interpretation) are below, so don't scroll down until you've completed the exercise.
- What was being studied?
- Who were the subjects of the study?
- How many subjects were there?
- Why were they doing the study? (What were they trying to learn?)
- How was the study conducted?
- What were the results of the study?
- What's the takeaway message?
What was being studied?
Answer: "The efficacy of a modular cognitive-behavioral therapy (CBT) protocol relative to treatment as usual (TAU) among children with high-functioning autism spectrum disorders (ASD) and clinically significant anxiety." In other words, what they were studying was whether cognitive behavioral therapy was better at helping children with autism and anxiety than regular therapy (treatment as usual).
Who were the subjects of the study?
Answer: "Children (7-11 years of age) with high-functioning ASD and clinically significant anxiety." It's important to note the specific details of the group being studied, such as the age and other qualities, because if the subjects studied are from a specific group, as these are, as opposed to being a representative sample of the population as a whole, that means that we don't know for sure if the results will apply to the population as a whole or just from that specific group. In this case, children ages 7-11 were evaluated, so if you were a 60 year old with high functioning autism, you can't know for sure based on this study if CBT therapy will work as well for you. Likewise, it was only studied on people with high functioning autism, so we don't necessarily know how it would work for people with low functioning autism.
How many subjects were there?
Answer: "A total of 45". This is important, because the larger the (random) sample, the more likely your results are to be reflective of the population as a whole. Imagine it this way - let's say you're trying to figure out what language most Americans speak. Completely at random, you pick one person from the US and see what language they speak. But what if the person you picked only speaks Russian, or Vietnamese, or some other non-English language? That's not representative of the dominant language of the United States as a whole, but if you're only picking one person, you might run into a problem like that.
How about if you pick five people at random? Well, the odds that they're reflective of the population norm (English speaking) just went up, but you still could have gotten unlucky and chosen five non-English speakers, which would give you the false impression that no one in the United States speaks English. How about if you chose 20? 30? 100? Each time your sample increases, so does the likelihood that your sample is going to look like the actual population you're trying to study (the United States) and not the product of bad luck in sampling. For that reason, we prefer to have larger sample sizes in studies. If the sample size is particularly low, then the odds that your study is an accurate reflection of most people is like are against you, and that study is weak evidence. A case study, which only studies one person, is a great example of this problem (which is why it's at the bottom of the evidence based medicine pyramid).
So how many subjects should a study have to be a good study? Well, the answer to that depends on a formula, but since I doubt most people want to do the formula each time they review a study, I recommend just following the rule of thumb of 30 or more participants, which this study meets.
How about if you pick five people at random? Well, the odds that they're reflective of the population norm (English speaking) just went up, but you still could have gotten unlucky and chosen five non-English speakers, which would give you the false impression that no one in the United States speaks English. How about if you chose 20? 30? 100? Each time your sample increases, so does the likelihood that your sample is going to look like the actual population you're trying to study (the United States) and not the product of bad luck in sampling. For that reason, we prefer to have larger sample sizes in studies. If the sample size is particularly low, then the odds that your study is an accurate reflection of most people is like are against you, and that study is weak evidence. A case study, which only studies one person, is a great example of this problem (which is why it's at the bottom of the evidence based medicine pyramid).
So how many subjects should a study have to be a good study? Well, the answer to that depends on a formula, but since I doubt most people want to do the formula each time they review a study, I recommend just following the rule of thumb of 30 or more participants, which this study meets.
Why were they doing the study? (AKA: What were they trying to learn?)
Answer: "To examine the efficacy of a modular cognitive-behavioral therapy (CBT) protocol relative to treatment as usual (TAU) among children with high-functioning autism spectrum disorders (ASD) and clinically significant anxiety."
In other words, they were doing the study to see is CBT therapy (which is a form of psychotherapy that you get from a psychologist, if you were interested in knowing) is more effective than treatment as usual to treat clinically significant anxiety.
In other words, they were doing the study to see is CBT therapy (which is a form of psychotherapy that you get from a psychologist, if you were interested in knowing) is more effective than treatment as usual to treat clinically significant anxiety.
How was the study conducted?
Answer: "A total of 45 children (7-11 years of age) with high-functioning ASD and clinically significant anxiety were randomized to receive 16 sessions of weekly CBT or TAU for an equivalent duration. After screening, assessments were conducted at baseline, post-treatment, and 3-month follow-up. Raters were blind to treatment condition."
There are some very important components to this that you should be looking out for. One, both groups were treated the same. If one group, for instance, had received a longer duration of therapy than the other, you wouldn't know if the difference between the groups you saw was because of the type of therapy or the length of the therapy (the third variable problem). And secondly, the assessments were conducted at baseline (meaning pre-treatment), immediately post-treatment, and after three months. What that should tell you is that the study sought to determine not only whether CBT was better than TAU for immediate effect, but also whether or not those effects held up after three months.
Perhaps the most important part of the methods is contained in the last sentence, "Raters were blind to treatment condition." What does that mean? Well, blinding in the context of a study means that the researchers don't know what condition the participant was in, which in this case means the people rating the child's anxiety didn't know whether the child was receiving CBT or TAU. That's important, because even if they're not consciously trying to, if researchers know which condition a participant is they could unconsciously be influencing results. If it's blind, researchers can't change results to reflect what they want the results to be accidentally.
There are some very important components to this that you should be looking out for. One, both groups were treated the same. If one group, for instance, had received a longer duration of therapy than the other, you wouldn't know if the difference between the groups you saw was because of the type of therapy or the length of the therapy (the third variable problem). And secondly, the assessments were conducted at baseline (meaning pre-treatment), immediately post-treatment, and after three months. What that should tell you is that the study sought to determine not only whether CBT was better than TAU for immediate effect, but also whether or not those effects held up after three months.
Perhaps the most important part of the methods is contained in the last sentence, "Raters were blind to treatment condition." What does that mean? Well, blinding in the context of a study means that the researchers don't know what condition the participant was in, which in this case means the people rating the child's anxiety didn't know whether the child was receiving CBT or TAU. That's important, because even if they're not consciously trying to, if researchers know which condition a participant is they could unconsciously be influencing results. If it's blind, researchers can't change results to reflect what they want the results to be accidentally.
What were the results of the study?
Answer: "Relative to usual care, CBT adapted for anxious youth with high-functioning ASD demonstrates large effects in reducing anxiety symptoms." Compared to TAU (usual care), CBT therapy for anxious children with high functioning autism was really effective at helping with anxiety. If you read something like this, it means they found some pretty convincing evidence (according to this study, at least) that what they're testing works.
What's the takeaway message?
This is the part where you evaluate everything you've already read and see the main results of the study. If you were seeking help with anxiety for a child on the autism spectrum, should you look for treatment as usual or CBT therapy? The answer, as I'm sure you know by now, is CBT therapy. I'd always recommend seeing if there's further literature out there (and in this case, there's actually a meta-analysis of multiple studies on the subject that would be even stronger evidence).
That said, from the abstract this study seems pretty robust and it's not advocating for anything potentially harmful (if it were, I would hold it to a far higher standard of proof and not act on its results without the advice of a trained medical professional), so overall I'd say if you had to choose between the two based off of this study I'd go for CBT therapy.
That said, from the abstract this study seems pretty robust and it's not advocating for anything potentially harmful (if it were, I would hold it to a far higher standard of proof and not act on its results without the advice of a trained medical professional), so overall I'd say if you had to choose between the two based off of this study I'd go for CBT therapy.
You did it!
Congratulations! You evaluated your first abstract and made it through the most dense article on the website! Now you should be able to do a much better job about evaluating scientific claims about autism.
Want to learn more about evidence based medicine? Check out one of these tutorials.
I'm going to take this one step further. If you ever have trouble with interpreting the results of a study about autism or a comorbid disorder, you can send it to me and I would be happy to assist you (for free, of course). The autism community needs more good science and I am happy to do my part to assist you towards that goal.
Want to learn more about evidence based medicine? Check out one of these tutorials.
I'm going to take this one step further. If you ever have trouble with interpreting the results of a study about autism or a comorbid disorder, you can send it to me and I would be happy to assist you (for free, of course). The autism community needs more good science and I am happy to do my part to assist you towards that goal.
