Verification pilot

Dunning–Kruger Reanalysis: The Scale, the Baseline, and the Error Curve

Verification pilot | Independent secondary analysis | Updated 18 November 2026

Result. We independently reanalysed 151 participants from a published Dunning–Kruger experiment. The classic negative association between baseline performance and signed self-assessment error reappears. However, the entire sample underestimated its absolute test score by an average of 15.28 points on the original chance-corrected 0–100 response scale (7.64 ordinary percentage points). After separating this overall offset from differences between participants, a marked skill–error gradient remains. The lowest baseline-skill quartile’s mean error alone has a wide interval that includes overestimation and underestimation.

Absolute score estimates and relative-rank estimates also behave differently. This distinction is well established in prior literature. Our result is an independent numerical verification and sensitivity audit, not a discovery of a new cognitive mechanism or a refutation of the Dunning–Kruger pattern. It also shows why the units, reference population and task difficulty matter when reading the usual curve.

Question

What survives when a Dunning–Kruger-shaped relationship is reported on the response scale actually used by participants, separated into a sample-wide mean error and a skill-dependent deviation, and compared with a different construct, perceived relative rank?

Competing explanations include differences in detecting errors, noisy observed skill and regression to the mean, task difficulty and the upper score boundary, prior beliefs or compressed self-ratings, and errors in estimating others’ performance. The present non-randomized observations do not identify which mechanism is causal.

Source and reconstruction

Primary source: McIntosh, Moore, Liu & Della Sala (2022); original OSF data and R code. There were 159 raw participant records; one duplicate and three ineligible non-students were removed, then four further participants at performance floor/ceiling were excluded, leaving 151 for analysis. Our source reconstruction follows these exclusions. The dataset has 48 baseline trials and 90 main-task trials per retained participant, with a 1–4 confidence rating after each main-task response and global self-estimates after the task.

Two score scales: Participants rated their absolute performance on an unusual 0–100 chance-corrected scale, where zero signifies 50% correct by chance on the two-choice task, and 100 signifies all answers correct. We reproduce the errors on that original response scale. To help readers interpret task accuracy, we also display ordinary percentages correct, calculated as 50 + (chance-corrected score)/2. That conversion necessarily halves all numerical absolute errors. It does not change their signs or relative ordering. Whether every participant understood the unusual scale exactly as intended was not independently tested, an important limitation for interpreting their self-estimates.

We formed descriptive quartiles of baseline skill, not main-task performance, with 38, 38, 37 and 38 participants. Baseline is measured before the main block, reducing one same-variable grouping problem, but it is a 48-trial measure and not a noise-free estimate of latent ability. Boundary ties are examined below. All analyses are observational and post hoc.

Result 1: absolute performance

For the complete sample, mean baseline skill was 55.66 and mean main-task performance 77.50 on the original scale. The mean self-estimate of main-task performance was 62.22. Thus, participants underestimated their main-task result by 15.28 original-scale points, or 7.64 ordinary percentage points. This overall intercept or offset must be considered before interpreting any particular quartile’s mean.

Baseline quartile n Actual correct, ordinary % Estimated correct, ordinary % Error, original 0–100 scale Error, ordinary pp
Lowest 38 76.1 76.0 −0.11 −0.05
Second 38 89.4 82.3 −14.03 −7.01
Third 37 92.6 81.0 −23.37 −11.68
Highest 38 97.0 85.1 −23.83 −11.92
Recalculation of McIntosh et al. (2022). Error = self-estimate minus observed score. The original-scale error is twice its numerical magnitude in ordinary percentage points. Quartiles are formed using the baseline task, not the task being estimated. Percentages and differences are rounded separately.

Shift versus gradient. After subtracting the complete sample’s mean error of −15.28 original-scale points, the lowest quartile is +15.17 points above that sample-wide offset, whereas the highest quartile is −8.55 below it. These centred deviations are approximately +7.59 and −4.28 ordinary percentage points, respectively. The negative skill–error pattern therefore remains. Near-zero uncentred error in one quartile is a property of the task, measurement and sample jointly, not proof that these people are especially well calibrated.

Continuous check, without discarding baseline values: An ordinary-least-squares regression of original-scale absolute error on original-scale baseline score gives a slope of −0.398 error points per baseline point (heteroskedasticity-robust HC3 95% CI −0.542 to −0.253; participant-bootstrap interval −0.546 to −0.256; n=151). Spearman ρ = −0.426. This reproduces the descriptive gradient, but does not correct baseline measurement error or identify an underlying psychological deficit. A split-half reliability estimate and an appropriate errors-in-variables model remain a separate, uncompleted test.

Uncertainty at the bottom: the lowest quartile’s signed mean is −0.11 original-scale points, with participant-bootstrap 95% CI −7.32 to +6.28 (or −3.66 to +3.14 ordinary pp). Individual errors are much larger than the mean: 22 of 38 overestimated, 14 underestimated, and two were exact; their mean absolute individual error is 7.74 ordinary percentage points. The confidence interval and the wide spread matter more than the near-zero point estimate.

Result 2: relative position

Participants also estimated their rank among other students. That answer concerns relative placement, not the percentage of questions answered correctly. Rank differences use percentile points; they are not directly comparable to either of the absolute-score error columns above. The self-reported reference population also differs from the available observed comparison group (see below).

Same baseline quartiles Observed sample percentile Self-estimated percentile Error, percentile points
Lowest 19.8 51.2 +31.4
Second 46.9 57.7 +10.9
Third 56.5 56.2 −0.3
Highest 79.0 68.7 −10.4
Authors’ percentile reference reconstructed from 155 eligible participants before the final four performance exclusions. The quartile rows contain the 151 retained participants.

In the lowest quartile, the mean rank estimate is near the sample’s midpoint despite a lower observed sample rank, while the group’s average signed absolute error includes zero. This is compatible with incorrect beliefs about others’ performance, but participants were not directly asked to estimate the typical other person’s score. Neither the two group means nor the rank answer alone identifies such beliefs or their cause. This distinction was developed in the prior literature, notably Moore & Healy (2008); it is not our novel mechanism.

Sensitivity checks and important limits

Where the 155-person reference comes from

The authors’ published R code calculates PERFORM_RANK = rank(COG_PERFORM)/length(COG_PERFORM)*100, before deleting four participants with extreme performance. With R’s default average-rank treatment of tied scores, the denominator is therefore 155, not 151 or 158. Our audit matches these author ranks.

Participants were asked to compare themselves with Edinburgh University students, yet the available 155-person reference includes alumni and people from other universities. Restricting the reference to the 120 retained current Edinburgh students changes the overall average relative-rank error from +7.94 to +10.52 percentile points. This does not resolve the mismatch: the 120 are still a convenience sample, not a representative benchmark for all Edinburgh students.

Ties, fixed order, and different task blocks

Twelve people tied at the lower-quartile baseline boundary. Across 8,000 random tie-breaking replications, the lowest quartile’s mean signed absolute error ranged from about −1.41 to +0.35 ordinary percentage points. The precise value −0.05 is sensitive to grouping, whereas a mean near zero is a more defensible description. This further argues against treating the exact decimal as a headline discovery.

Baseline and main task were not identical repeated measurements: the baseline had 48 trials and the main task had 90, in a fixed order. The mean increased from 55.66 to 77.50 on the chance-corrected scale, a gain of 21.84 original points, equivalent to 10.92 ordinary percentage points correct. This could reflect practice, unequal difficulty, or both. It is not a randomized estimate of learning. Consequently, a baseline quartile need not represent the same ability quartile during the main block.

Scale boundaries and interpretation

The main block was easy for this selected sample: the highest baseline quartile answered about 97% correctly. The resulting ceiling limits how far those participants could overestimate actual accuracy. A task-wide shift in self-estimation can move the whole error curve up or down while preserving a skill-dependent gradient; the data do not establish that task difficulty alone caused the observed −15.28-point offset. The unusual 0–100 chance-corrected response format introduces another construct-validity risk: a participant who misunderstands the instructions can exhibit apparent miscalibration without an independently identified metacognitive problem.

Why a negative slope is not a mechanism

When signed error is self-rating S minus objective score X, the covariance identity Cov(S−X, X) = Cov(S, X) − Var(X) shows how using the outcome in its own error can produce a negative association. Grouping by an earlier baseline reduces that exact algebraic overlap, but does not recover latent skill or remove all regression, noise and selection. Even a Bayes-optimal self-estimate based on noisy private information can show a slope below one when plotted against actual ability. A continuous regression of error on baseline, reported above, is therefore more informative than quartile selection but still not a causal test of defective metacognition. The follow-up needed is an independent reliability estimate and an explicitly specified errors-in-variables or latent-skill analysis.

Context: an old distinction, not a new debunking

Moore & Healy (2008) distinguished overestimation, overplacement and overprecision. Krueger & Mueller (2002) argued that better-than-average judgments and regression can generate Dunning–Kruger-shaped patterns. Burson, Larrick & Klayman (2006) found that changes in task difficulty can change the direction of relative-rank miscalibration. The original Kruger & Dunning (1999) experiments included more than quartile curves: their logic-training study also examined whether improving skill changed self-assessment. Any fair assessment of the dual-burden mechanism must consider that evidence, not just a diagram.

McIntosh and colleagues themselves reported evidence against a strong dual-burden explanation in their task, while noting that less skilled participants showed poorer local discrimination between correct and incorrect answers. Our calculations do not overturn their article. They give a specific, numerically checked illustration of why absolute prediction, relative placement and trialwise metacognitive discrimination must be reported separately.

Reproducibility and scope

Design: post hoc independent secondary analysis of previously published data; no new participants, randomization or preregistration by the Institute. Sample: 151 retained participants after the authors’ exclusions; 155 form the authors’ rank denominator. Absolute signed error: self-estimated score minus actual main-task score, primarily reported on the original chance-corrected 0–100 scale. Ordinary percentage correct: 50 + corrected_score/2; converting the error to ordinary percentage points divides it by two. Relative-rank error: self-estimated rank minus observed sample rank in percentile points. No standardized common magnitude follows from comparing these units.

Calculations: descriptive quartile means use independent baseline ranking with deterministic tie resolution; participant-bootstrap percentile intervals and tie sensitivity are reported; the new continuous linear regression uses HC3 robust uncertainty and a 6,000-resample participant bootstrap, with a fixed resampling seed. It has not been corrected for baseline measurement error. The underlying individual-level data were checked against our prior verified archive, including an original archive SHA-256 of 49bb3087f13f9090cff742e81da20340739902e2b626625a900850694077a570 for the Institute’s private participant-summary and calculation ZIP. These extra checks revise interpretation, not the source experimental design.

Non-primary sign check: 61 of 151 participants both underestimated absolute score and overestimated relative rank. Given 117 absolute underestimates and 95 rank overplacements, a purely hypothetical independent-sign benchmark predicts 73.6, not 61. The two scores and errors share underlying task information, so this benchmark is neither a plausible causal null model nor evidence of a new mechanism. We do not use the joint count as a finding.

Reproduce this verification: download the Institute’s Python code and aggregate reference package (ZIP, 7,413 bytes). It includes reproduce.py, a README, dependency list, expected aggregate outputs, a file manifest with SHA-256 hashes, and an MIT license for the Institute’s own code. Package SHA-256: 2605385908a00e6fbcc1a07be2a4d8ceee1d26332fbc64cd20a950bc70c61519. The script reads the original authors’ SSK_full.csv, which readers must download separately from McIntosh et al. (2022) on OSF; no individual participant data or original authors’ scripts are redistributed in our ZIP. This release reproduces the main descriptive tables and continuous regression, but does not claim to independently validate the questionnaire or identify a psychological mechanism. The broader research archives and exploratory analyses remain private.

Conclusion: the original Dunning–Kruger-shaped association is reproducible in this dataset. Its lowest baseline quartile happens to have a nearly zero mean signed absolute error, but that value is imprecise and depends on a large sample-wide underestimation offset, grouping choices and the measurement format. Large relative-rank errors describe another construct with an imperfect comparison population. Our verification supports separating the scales and testing rival mechanisms, not declaring a discovery, a debunking, or a new explanation of participants’ beliefs.

References

  1. Kruger J, Dunning D. (1999). Unskilled and unaware of it. Journal of Personality and Social Psychology. DOI.
  2. Krueger J, Mueller RA. (2002). Unskilled, unaware, or both? Journal of Personality and Social Psychology. PubMed.
  3. Burson KA, Larrick RP, Klayman J. (2006). Skilled or unskilled, but still unaware of it. Journal of Personality and Social Psychology. DOI.
  4. Moore DA, Healy PJ. (2008). The trouble with overconfidence. Psychological Review. DOI.
  5. McIntosh RD, Moore AB, Liu Y, Della Sala S. (2022). Skill and self-knowledge: empirical refutation of the dual-burden account of the Dunning–Kruger effect. Royal Society Open Science 9, 191727. DOI; data and code.