>

>

What is effect size and why reviewers care

What is effect size and why reviewers care

Princeton Journal of Pre-Collegiate Research

What is effect size and why reviewers care

If you have ever submitted a manuscript only to receive reviewer comments asking about effect sizes, you are not alone. Understanding what is effect size and why reviewers care about it is essential for any researcher looking to publish in peer-reviewed journals. Effect size is a statistical measure that quantifies the magnitude of a relationship or difference between groups, going beyond the simple yes-or-no answer provided by p-values. In this guide, we will break down everything you need to know about effect sizes, how to calculate them, and why they have become a non-negotiable element of modern research reporting.

What is Effect Size and Why Reviewers Care: The Basic Definition

An effect size is a numerical value that describes the practical magnitude of a research finding. While a p-value tells you whether a result is statistically significant, an effect size tells you how large or how meaningful that result actually is. This distinction is critical because a study with a very large sample size can produce a statistically significant result even when the actual difference between groups is trivially small.

Consider a clinical trial testing a new medication. The trial might show a statistically significant reduction in blood pressure (p < 0.05), but if the effect size is tiny, the reduction may be so small that it has no real-world clinical relevance. Effect sizes give readers and reviewers the information they need to judge whether a finding matters in practice, not just in theory.

Common types of effect size measures include:

  • Cohen's d – used for comparing two means

  • Pearson's r – used for correlations

  • Eta-squared (η²) – used in ANOVA designs

  • Odds ratio – used in logistic regression and clinical research

  • Hedges' g – a corrected version of Cohen's d for small samples

The Problem with P-Values Alone

For decades, researchers relied almost exclusively on p-values to determine whether their findings were worth reporting. A result with p < 0.05 was considered significant, while anything above that threshold was dismissed. This approach created several serious problems in the scientific literature.

First, p-values are heavily influenced by sample size. With enough participants, almost any trivial difference will become statistically significant. Second, p-values tell you nothing about the direction or size of an effect. Two studies could both report p = 0.03 while describing effects of completely different magnitudes. Third, the binary significant/non-significant framework encourages publication bias, where only positive results get published.

Major statistical organizations, including the American Statistical Association, have issued statements urging researchers to move beyond p-values and report effect sizes alongside confidence intervals. Many top journals now require effect sizes as a condition of publication, which is precisely why reviewers flag manuscripts that omit them.

How to Calculate Common Effect Sizes

Calculating effect sizes is straightforward once you know which measure is appropriate for your study design.

Cohen's d for Mean Differences

Cohen's d is calculated by dividing the difference between two group means by the pooled standard deviation. The formula is:

d = (M₁ - M₂) / SD_pooled

Jacob Cohen proposed the following benchmarks for interpreting d values:

  • Small effect: d = 0.2

  • Medium effect: d = 0.5

  • Large effect: d = 0.8

However, these benchmarks are field-dependent. A d of 0.3 might be considered large in educational research but small in pharmacology. Always interpret effect sizes in the context of your specific field.

Pearson's r for Correlations

When examining the relationship between two continuous variables, Pearson's r serves as both a correlation coefficient and an effect size measure. Cohen's benchmarks for r are:

  • Small effect: r = 0.1

  • Medium effect: r = 0.3

  • Large effect: r = 0.5

Eta-Squared for ANOVA

Eta-squared (η²) represents the proportion of total variance in the dependent variable that is accounted for by the independent variable. Values range from 0 to 1, with higher values indicating larger effects. Partial eta-squared (ηp²) is often preferred in factorial designs because it controls for variance explained by other factors in the model.

What is Effect Size and Why Reviewers Care: Reporting Standards

Understanding what is effect size and why reviewers care becomes especially clear when you look at modern journal reporting guidelines. The APA Publication Manual (7th edition) explicitly states that effect sizes should be reported for all primary outcomes. The CONSORT guidelines for clinical trials, STROBE for observational studies, and PRISMA for systematic reviews all include effect size reporting as a core requirement.

When reviewers evaluate a manuscript, they are looking for several things related to effect sizes:

  1. Presence – Is an effect size reported at all?

  2. Appropriateness – Is the correct type of effect size used for the study design?

  3. Interpretation – Does the author explain what the effect size means in practical terms?

  4. Confidence intervals – Are confidence intervals provided around the effect size estimate?

  5. Consistency – Do the effect sizes align with the conclusions drawn in the discussion section?

A manuscript that reports only p-values without effect sizes sends a signal to reviewers that the authors may not fully understand modern statistical reporting standards. This can undermine confidence in the entire paper, even if the underlying research is sound.

Effect Sizes in Meta-Analysis and Systematic Reviews

Effect sizes play an especially critical role in meta-analyses and systematic reviews. These study types aggregate findings from multiple primary studies, and effect sizes provide the common currency that makes this aggregation possible. Without standardized effect sizes, it would be impossible to combine results across studies that used different scales, sample sizes, or measurement instruments.

When a systematic reviewer searches the literature and finds that your study does not report an effect size, they face a difficult choice: attempt to calculate it from the data you provided, contact you directly, or exclude your study from the meta-analysis entirely. Studies excluded from meta-analyses have less scientific impact, which is another practical reason why reporting effect sizes matters for the long-term influence of your work.

Forest plots, the signature visualization of meta-analyses, display effect sizes and their confidence intervals for each included study. A well-reported effect size in your original study makes it easier for future researchers to include your work in these influential syntheses.

Practical Tips for Reporting Effect Sizes in Your Manuscript

Now that you understand the theory, here are actionable steps to ensure your effect size reporting satisfies reviewers:

Report Effect Sizes for All Primary and Secondary Outcomes

Do not limit effect size reporting to your main hypothesis. Include effect sizes for secondary outcomes, subgroup analyses, and any exploratory findings. This gives reviewers a complete picture of your results and demonstrates statistical sophistication.

Always Include Confidence Intervals

A point estimate of an effect size without a confidence interval is incomplete. Confidence intervals communicate the precision of your estimate and allow readers to assess whether the true effect could plausibly be negligible. Report 95% confidence intervals as a minimum, and consider 90% intervals in exploratory research contexts.

Interpret Effect Sizes in Context

Do not simply state that an effect is small, medium, or large according to Cohen's benchmarks. Explain what the effect size means for your specific population, outcome, and field. If a small effect size translates to thousands of lives saved at a population level, say so. If a large effect size corresponds to a difference that patients would not notice in daily life, acknowledge that too.

Use Software to Calculate Effect Sizes Accurately

Most statistical software packages can calculate effect sizes automatically. R packages such as effectsize and rstatix make this straightforward. SPSS provides partial eta-squared in ANOVA output. G*Power can calculate effect sizes for power analyses. Avoid calculating effect sizes by hand unless you are confident in the formulas, as errors are common and can undermine your credibility with reviewers.

Common Mistakes That Draw Reviewer Criticism

Even researchers who understand effect sizes sometimes make mistakes in reporting them. The most common errors include:

  • Confusing statistical significance with practical significance and using one to justify the other

  • Reporting effect sizes without confidence intervals

  • Using eta-squared instead of partial eta-squared in factorial ANOVA designs, which inflates the apparent effect size

  • Applying Cohen's benchmarks without considering field-specific norms

  • Failing to report effect sizes for non-significant results, which are just as informative as significant ones

  • Inconsistency between the effect sizes reported in tables and those discussed in the text

Reviewers who specialize in quantitative methods will catch these errors quickly. Addressing them before submission saves time and increases your chances of acceptance.

The Future of Effect Size Reporting

The emphasis on effect sizes is not a passing trend. As the replication crisis continues to reshape scientific practice, journals and funding agencies are placing increasing demands on transparent, complete statistical reporting. Pre-registration of studies now often requires researchers to specify expected effect sizes for power calculations before data collection begins. Open science frameworks encourage sharing raw data so that effect sizes can be independently verified.

Emerging reporting standards are also moving toward equivalence testing and minimum effect sizes of interest (SESOI), which require researchers to define in advance how large an effect must be to matter practically. These developments make a thorough understanding of effect sizes more important than ever for researchers at every career stage.

Conclusion

Understanding what is effect size and why reviewers care is no longer optional for researchers who want to publish in competitive journals. Effect sizes provide the practical context that p-values cannot, enable future meta-analyses, and demonstrate that you understand modern statistical reporting standards. By calculating appropriate effect sizes, reporting them with confidence intervals, and interpreting them meaningfully in the context of your field, you will produce manuscripts that satisfy reviewers and contribute more effectively to the scientific literature. Make effect size reporting a standard part of your research workflow, and you will find that reviewer comments on this topic become a thing of the past.


Students who apply these principles correctly produce results that are both defensible under peer review and genuinely informative. The methodology is accessible, the tools are free, and the analytical standard is well-established. If your research uses is ready for independent evaluation, submit it to PJPCR at princeton-jpcr.org/submit.

Read More

Diagram illustrating axion particle detection apparatus used in dark matter research experiments

Axions and the Search for Dark Matter, Explained for Students

By

Princeton Journal of Pre-Collegiate Research

Read more

Conceptual diagram of axion dark matter detection apparatus with magnetic field coils and microwave cavity resonator for student research

Axions and dark matter: 10 Research Questions and Experiment Ideas for Students

By

Princeton Journal of Pre-Collegiate Research

Read more

Conceptual illustration of axion particles and dark matter distribution in a galactic halo, relevant to particle physics and cosmology research

Axions and dark matter: 12 Questions Students Ask, Answered

By

Princeton Journal of Pre-Collegiate Research

Read more

Diagram illustrating the distance from the Sun to Earth and Mars with labeled orbital paths in the solar system

Distance From the Sun to Earth (and Mars): Solar System Distances Explained

By

Princeton Journal of Pre-Collegiate Research

Read more

Diagram showing scaled distances between planets in the solar system for student research and educational experiments

Distances in the solar system: 10 Research Questions and Experiment Ideas for Students

By

Princeton Journal of Pre-Collegiate Research

Read more

Diagram illustrating the relative distances between planets in the solar system, with scale markers in astronomical units

Distances in the solar system: 12 Questions Students Ask, Answered

By

Princeton Journal of Pre-Collegiate Research

Read more

False-color map of Mars crustal magnetic field anomalies detected by the Mars Global Surveyor magnetometer, showing striped magnetic signatures in the southern highlands

Does Mars Have a Magnetic Field? What We Know and How We Know

By

Princeton Journal of Pre-Collegiate Research

Read more

The Martian magnetic field: 10 Research Questions and Experiment Ideas for Students

The Martian magnetic field: 10 Research Questions and Experiment Ideas for Students

By

Princeton Journal of Pre-Collegiate Research

Read more

Illustration of Mars with crustal magnetic field lines and solar wind interaction depicted in scientific diagram style

The Martian magnetic field: 12 Questions Students Ask, Answered

By

Princeton Journal of Pre-Collegiate Research

Read more

Diagram illustrating rolling friction forces acting on a wheel in contact with a flat surface, with labeled vectors for normal force and resistive torque

Rolling Friction Explained: Definition, Examples and Experiments for Students

By

Princeton Journal of Pre-Collegiate Research

Read more

High school student measuring rolling friction of a cylinder on an inclined plane in a physics laboratory setting

Rolling friction: 10 Research Questions and Experiment Ideas for Students

By

Princeton Journal of Pre-Collegiate Research

Read more

Diagram illustrating rolling friction forces acting on a sphere rolling across a flat surface, with labeled vectors for normal force, friction, and deformation

Rolling friction: 12 Questions Students Ask, Answered

By

Princeton Journal of Pre-Collegiate Research

Read more

High school student conducting a controlled friction experiment on an inclined plane with a wooden block and spring scale

How to Find the Coefficient of Kinetic Friction: A Student Experiment Guide

By

Princeton Journal of Pre-Collegiate Research

Read more

High school student measuring the coefficient of kinetic friction using a wooden block and spring scale on an inclined plane in a laboratory setting

The coefficient of kinetic friction: 10 Research Questions and Experiment Ideas for Students

By

Princeton Journal of Pre-Collegiate Research

Read more

The coefficient of kinetic friction: 12 Questions Students Ask, Answered

The coefficient of kinetic friction: 12 Questions Students Ask, Answered

By

Princeton Journal of Pre-Collegiate Research

Read more

Diagram of standing wave harmonics on a vibrating string showing nodes and antinodes at fundamental and overtone frequencies

Harmonics and Standing Waves: What They Are and How to Test Them

By

Princeton Journal of Pre-Collegiate Research

Read more

High school student conducting a standing wave experiment with a vibrating string apparatus in a physics laboratory

Harmonics and standing waves: 10 Research Questions and Experiment Ideas for Students

By

Princeton Journal of Pre-Collegiate Research

Read more

Diagram illustrating harmonic frequencies and standing wave nodes and antinodes on a vibrating string in a physics laboratory setting

Harmonics and standing waves: 12 Questions Students Ask, Answered

By

Princeton Journal of Pre-Collegiate Research

Read more

Mathematical proof of the AM-GM inequality written on a chalkboard with algebraic notation and inequality symbols

AM-GM Inequality: Proof, Examples and Olympiad Uses

By

Princeton Journal of Pre-Collegiate Research

Read more

Educator reviewing a student research manuscript in an academic setting with scientific journals and data charts on the desk

For Educators: Supporting Student Research and Publication

By

Princeton Journal of Pre-Collegiate Research

Read more

A high school teacher reviewing a student's research manuscript at a desk covered with scientific papers and reference materials

How Teachers Can Support a Student’s First Publication

By

Princeton Journal of Pre-Collegiate Research

Read more

Princeton, New Jersey, United States
Published and Managed by The Princeton Journal of Precollegiate Scholarship Inc.
ISSN: 3143-8423
DOI: 10.67698

Copyright © Princeton Journal of Pre-Collegiate Research. All rights reserved

PJPCR is independently operated and is not affiliated with Princeton University or any of its colleges, departments or programs.

Princeton, New Jersey, United States
Published and Managed by The Princeton Journal of Precollegiate Scholarship Inc.
ISSN: 3143-8423
DOI: 10.67698

Copyright © Princeton Journal of Pre-Collegiate Research. All rights reserved

PJPCR is independently operated and is not affiliated with Princeton University or any of its colleges, departments or programs.

Princeton, New Jersey, United States
Published and Managed by The Princeton Journal of Precollegiate Scholarship Inc.
ISSN: 3143-8423
DOI: 10.67698

Copyright © Princeton Journal of Pre-Collegiate Research. All rights reserved

PJPCR is independently operated and is not affiliated with Princeton University or any of its colleges, departments or programs.