What is Effect Size and Why Reviewers Care About It in Research

If you have ever submitted a manuscript only to receive reviewer comments asking about effect sizes, you are not alone. Understanding what is effect size and why reviewers care about it is essential for any researcher looking to publish in peer-reviewed journals. Effect size is a statistical measure that quantifies the magnitude of a relationship or difference between groups, going beyond the simple yes-or-no answer provided by p-values. In this guide, we will break down everything you need to know about effect sizes, how to calculate them, and why they have become a non-negotiable element of modern research reporting.

What is Effect Size and Why Reviewers Care: The Basic Definition

An effect size is a numerical value that describes the practical magnitude of a research finding. While a p-value tells you whether a result is statistically significant, an effect size tells you how large or how meaningful that result actually is. This distinction is critical because a study with a very large sample size can produce a statistically significant result even when the actual difference between groups is trivially small.

Consider a clinical trial testing a new medication. The trial might show a statistically significant reduction in blood pressure (p < 0.05), but if the effect size is tiny, the reduction may be so small that it has no real-world clinical relevance. Effect sizes give readers and reviewers the information they need to judge whether a finding matters in practice, not just in theory.

Common types of effect size measures include:

• Cohen's d – used for comparing two means

• Pearson's r – used for correlations

• Eta-squared (η²) – used in ANOVA designs

• Odds ratio – used in logistic regression and clinical research

• Hedges' g – a corrected version of Cohen's d for small samples

The Problem with P-Values Alone

For decades, researchers relied almost exclusively on p-values to determine whether their findings were worth reporting. A result with p < 0.05 was considered significant, while anything above that threshold was dismissed. This approach created several serious problems in the scientific literature.

First, p-values are heavily influenced by sample size. With enough participants, almost any trivial difference will become statistically significant. Second, p-values tell you nothing about the direction or size of an effect. Two studies could both report p = 0.03 while describing effects of completely different magnitudes. Third, the binary significant/non-significant framework encourages publication bias, where only positive results get published.

Major statistical organizations, including the American Statistical Association, have issued statements urging researchers to move beyond p-values and report effect sizes alongside confidence intervals. Many top journals now require effect sizes as a condition of publication, which is precisely why reviewers flag manuscripts that omit them.

How to Calculate Common Effect Sizes

Calculating effect sizes is straightforward once you know which measure is appropriate for your study design.

Cohen's d for Mean Differences

Cohen's d is calculated by dividing the difference between two group means by the pooled standard deviation. The formula is:

d = (M₁ - M₂) / SD_pooled

Jacob Cohen proposed the following benchmarks for interpreting d values:

• Small effect: d = 0.2

• Medium effect: d = 0.5

• Large effect: d = 0.8

However, these benchmarks are field-dependent. A d of 0.3 might be considered large in educational research but small in pharmacology. Always interpret effect sizes in the context of your specific field.

Pearson's r for Correlations

When examining the relationship between two continuous variables, Pearson's r serves as both a correlation coefficient and an effect size measure. Cohen's benchmarks for r are:

• Small effect: r = 0.1

• Medium effect: r = 0.3

• Large effect: r = 0.5

Eta-Squared for ANOVA

Eta-squared (η²) represents the proportion of total variance in the dependent variable that is accounted for by the independent variable. Values range from 0 to 1, with higher values indicating larger effects. Partial eta-squared (ηp²) is often preferred in factorial designs because it controls for variance explained by other factors in the model.

What is Effect Size and Why Reviewers Care: Reporting Standards

Understanding what is effect size and why reviewers care becomes especially clear when you look at modern journal reporting guidelines. The APA Publication Manual (7th edition) explicitly states that effect sizes should be reported for all primary outcomes. The CONSORT guidelines for clinical trials, STROBE for observational studies, and PRISMA for systematic reviews all include effect size reporting as a core requirement.

When reviewers evaluate a manuscript, they are looking for several things related to effect sizes:

1. Presence – Is an effect size reported at all?

2. Appropriateness – Is the correct type of effect size used for the study design?

3. Interpretation – Does the author explain what the effect size means in practical terms?

4. Confidence intervals – Are confidence intervals provided around the effect size estimate?

5. Consistency – Do the effect sizes align with the conclusions drawn in the discussion section?

A manuscript that reports only p-values without effect sizes sends a signal to reviewers that the authors may not fully understand modern statistical reporting standards. This can undermine confidence in the entire paper, even if the underlying research is sound.

Effect Sizes in Meta-Analysis and Systematic Reviews

Effect sizes play an especially critical role in meta-analyses and systematic reviews. These study types aggregate findings from multiple primary studies, and effect sizes provide the common currency that makes this aggregation possible. Without standardized effect sizes, it would be impossible to combine results across studies that used different scales, sample sizes, or measurement instruments.

When a systematic reviewer searches the literature and finds that your study does not report an effect size, they face a difficult choice: attempt to calculate it from the data you provided, contact you directly, or exclude your study from the meta-analysis entirely. Studies excluded from meta-analyses have less scientific impact, which is another practical reason why reporting effect sizes matters for the long-term influence of your work.

Forest plots, the signature visualization of meta-analyses, display effect sizes and their confidence intervals for each included study. A well-reported effect size in your original study makes it easier for future researchers to include your work in these influential syntheses.

Practical Tips for Reporting Effect Sizes in Your Manuscript

Now that you understand the theory, here are actionable steps to ensure your effect size reporting satisfies reviewers:

Report Effect Sizes for All Primary and Secondary Outcomes

Do not limit effect size reporting to your main hypothesis. Include effect sizes for secondary outcomes, subgroup analyses, and any exploratory findings. This gives reviewers a complete picture of your results and demonstrates statistical sophistication.

Always Include Confidence Intervals

A point estimate of an effect size without a confidence interval is incomplete. Confidence intervals communicate the precision of your estimate and allow readers to assess whether the true effect could plausibly be negligible. Report 95% confidence intervals as a minimum, and consider 90% intervals in exploratory research contexts.

Interpret Effect Sizes in Context

Do not simply state that an effect is small, medium, or large according to Cohen's benchmarks. Explain what the effect size means for your specific population, outcome, and field. If a small effect size translates to thousands of lives saved at a population level, say so. If a large effect size corresponds to a difference that patients would not notice in daily life, acknowledge that too.

Use Software to Calculate Effect Sizes Accurately

Most statistical software packages can calculate effect sizes automatically. R packages such as effectsize and rstatix make this straightforward. SPSS provides partial eta-squared in ANOVA output. G*Power can calculate effect sizes for power analyses. Avoid calculating effect sizes by hand unless you are confident in the formulas, as errors are common and can undermine your credibility with reviewers.

Common Mistakes That Draw Reviewer Criticism

Even researchers who understand effect sizes sometimes make mistakes in reporting them. The most common errors include:

• Confusing statistical significance with practical significance and using one to justify the other

• Reporting effect sizes without confidence intervals

• Using eta-squared instead of partial eta-squared in factorial ANOVA designs, which inflates the apparent effect size

• Applying Cohen's benchmarks without considering field-specific norms

• Failing to report effect sizes for non-significant results, which are just as informative as significant ones

• Inconsistency between the effect sizes reported in tables and those discussed in the text

Reviewers who specialize in quantitative methods will catch these errors quickly. Addressing them before submission saves time and increases your chances of acceptance.

The Future of Effect Size Reporting

The emphasis on effect sizes is not a passing trend. As the replication crisis continues to reshape scientific practice, journals and funding agencies are placing increasing demands on transparent, complete statistical reporting. Pre-registration of studies now often requires researchers to specify expected effect sizes for power calculations before data collection begins. Open science frameworks encourage sharing raw data so that effect sizes can be independently verified.

Emerging reporting standards are also moving toward equivalence testing and minimum effect sizes of interest (SESOI), which require researchers to define in advance how large an effect must be to matter practically. These developments make a thorough understanding of effect sizes more important than ever for researchers at every career stage.

Conclusion

Understanding what is effect size and why reviewers care is no longer optional for researchers who want to publish in competitive journals. Effect sizes provide the practical context that p-values cannot, enable future meta-analyses, and demonstrate that you understand modern statistical reporting standards. By calculating appropriate effect sizes, reporting them with confidence intervals, and interpreting them meaningfully in the context of your field, you will produce manuscripts that satisfy reviewers and contribute more effectively to the scientific literature. Make effect size reporting a standard part of your research workflow, and you will find that reviewer comments on this topic become a thing of the past.