Statistics Done Wrong
Alex Reinhart
33 min
10 Key Points
4.3 RateWhat's inside?
Dive into the common mistakes in statistical analysis and learn how to avoid them, improving your understanding and application of statistics in any field.
You'll learn
1. Watch out for common stats blunders!2. Making sense of stats results.3. Why planning your experiment matters in stats.4. Spotting and fixing stats misuse.5. The sneaky role of bias in stats.6. Up your stats game!Key points
01Understanding and Correcting Misconceptions about P-Values
P-values. You've probably heard of them, especially if you've ever dabbled in the world of statistics or research. They're like the secret sauce that adds credibility to a study, the magic number that can make or break a hypothesis. But here's the thing: p-values are often misunderstood and misused, leading to flawed research outcomes and false conclusions. Let's start with the basics. A p-value is not the probability that the null hypothesis is true, nor is it the probability that your results occurred by chance. These are common misconceptions that can lead to incorrect interpretations of research findings. Instead, a p-value is a measure of how incompatible your data is with the null hypothesis. It's like a weighing scale that measures the strength of evidence against the null hypothesis. The lower the p-value, the stronger the evidence against the null hypothesis. But here's where things get tricky. P-values are often misused in research. For instance, some researchers might interpret a p-value of 0.05 as strong evidence against the null hypothesis, when in fact, it only indicates that the observed data would be quite unusual if the null hypothesis were true. This can lead to overconfidence in research findings and the dismissal of potentially important results. Another common mistake is the misuse of p-values as a measure of effect size or importance. A small p-value does not necessarily mean that the effect is large or important, and a large p-value does not mean that the effect is small or unimportant. P-values are simply a measure of evidence against the null hypothesis, not a measure of the size or importance of the effect. So, how should p-values be correctly used in research? First and foremost, p-values should be used in conjunction with other statistical measures, such as confidence intervals and effect sizes. They should also be interpreted within the context of the research question and the study design. For instance, a small p-value might be more meaningful in a large, well-designed study than in a small, poorly designed study. Correcting misconceptions about p-values is crucial for improving the quality of research. It's important to understand that p-values are not a definitive proof of a hypothesis, but rather a measure of evidence against the null hypothesis. They have limitations and should be used appropriately. In conclusion, understanding and correctly using p-values is essential for conducting and interpreting research. It's not just about getting that magic number; it's about understanding what that number means and how it fits into the bigger picture of your research. So, the next time you come across a p-value, remember: it's not just a number, it's a tool for understanding your data. Use it wisely.
02Understanding the Misconceptions of Statistical Significance
Ever wondered why some research findings that seem important are dismissed, while others that seem trivial are hailed as groundbreaking? The answer often lies in the misunderstood and misused concept of statistical significance. Statistical significance is like a referee in a football match. It doesn't tell us which team played better or scored the most goals. It simply tells us whether the game's result is likely due to the teams' skills or just a fluke. Similarly, statistical significance doesn't tell us if a research finding is important or relevant. It merely tells us if the finding is likely due to the studied effect or just a random chance. However, many people misunderstand statistical significance as a definitive measure of a finding's importance. They think that if a finding is statistically significant, it must be important. But that's not the case. A finding can be statistically significant but practically insignificant. For example, a drug might significantly reduce the risk of a disease, but if the reduction is only by 0.01%, is it really important? This misunderstanding leads to an overreliance on statistical significance in research. Researchers often dismiss findings that are not statistically significant, even if they are practically important. They also overemphasize findings that are statistically significant, even if they are practically insignificant. For instance, in the book, Reinhart discusses a study that found a statistically significant link between jelly beans and acne. But the effect size was so small that it was practically insignificant. This brings us to the concept of effect size. Think of effect size as the number of goals scored in a football match. It tells us the magnitude of the difference or the effect. A large effect size means a big difference or a strong effect, while a small effect size means a small difference or a weak effect. Effect size is a more meaningful measure than statistical significance alone because it tells us not just whether an effect exists, but also how big the effect is. Practical significance is another important concept. It's like asking whether winning a football match by one goal is important. It depends on the context. If it's a friendly match, maybe not. But if it's a World Cup final, definitely yes. Similarly, whether a research finding is practically significant depends on the context. For example, a small reduction in disease risk might not be practically significant for a healthy person, but it might be for a high-risk person. The misuse of statistical significance can lead to erroneous conclusions and misguided decisions. For example, Reinhart discusses a study that found a statistically significant increase in cancer risk for people living near power lines. But the increase was so small that it was practically insignificant. Yet, the study caused unnecessary fear and panic. Reinhart recommends using statistical significance as one of many tools in research. It's important to also consider effect size, practical significance, and other statistical measures. So, next time you read a research finding, don't just ask if it's statistically significant. Ask also how big the effect is and whether it's practically significant. And remember, statistical significance is just the referee. It doesn't decide who wins the match.
Continue reading with LeapAhead app
Full summary is waiting for you in the app
03Understanding the Problem of Multiple Comparisons in Statistical Analysis
04Understanding and Minimizing Selection Bias in Studies
05Understanding and Preventing Overfitting in Statistical Modeling
06Understanding Challenges in Establishing Causality in Research
07Understanding Statistical Power in Research
08"Misuse of Regression Analysis in Research: Common Mistakes and Correct Interpretation Strategies"
09What's data dredging all about?
10Conclusion
About Alex Reinhart
Alex Reinhart is a statistics instructor and PhD candidate in Statistics at Carnegie Mellon University. He is known for his work in statistical education, particularly his book "Statistics Done Wrong: The Woefully Complete Guide".