A completed workflow is not a verified scientific finding. In our experimental AI research laboratory, two projects marked completed ended with the word UNRESOLVED. Elsewhere, agents produced reproducible calculations but gave one statistical shortcut more authority than it deserved. Those are different failures, and understanding the difference is the point of this audit.
Research Note · The Null Institute · 12 October 2026 · Ilya Nikitin, Ph.D. · Version 1.0
Study type: retrospective, author-led technical and methodological audit. AI-assisted analysis and drafting; not independently peer-reviewed. Source-level replication remains incomplete.
Abstract
We audited archived records from an experimental autonomous research laboratory built to turn research questions into traceable evidence, computations and reports under small budgets. Native PostgreSQL queries confirmed 48 project registrations and 42 completed workflow states in the latest system; two of the recovered completed reports explicitly returned UNRESOLVED. We recovered 42 written final outputs, and independently checked the integrity of 103 registered computational objects from four World Happiness Report investigations. Recalculations from preserved intermediate tables reproduced key country counts and correlations, but also exposed an unjustified interpretation of rank uncertainty: in 25 of 147 cases, an interval-overlap heuristic labelled “conservative” failed to encompass the published rank interval. A sensitivity check further showed that the relationship between life evaluation and emotional experience varies with the chosen affect construct. These results document genuine research-related capability alongside gaps in outcome evaluation and statistical interpretation. They do not establish an overall error rate, a performance advantage over simpler workflows, or independently validated scientific discoveries.
Why audit a research lab that seemed to work?
The system was designed as more than a writing assistant. It divided a question into research tasks, retrieved material, registered claims and objections, commissioned calculations, recorded costs and produced a final answer. Success looked measurable: tasks finished, artifacts accumulated, and reports appeared.
But a laboratory has two kinds of outputs. One is operational: a job ran to completion. The other is epistemic: a conclusion is supported, reproducible, appropriately qualified and, where claimed, genuinely new. A database status cannot substitute for the second. Our audit asked what the surviving records could actually establish.
What survived, and how we checked it
We preserved two stopped PostgreSQL 16 data directories. They contained three logical databases representing different development branches: institute, institute_lab2 and institute_v02. We did not pool their projects: some research history and testing may overlap. On 12 October 2026, native SQL queries against isolated copies of the database directories confirmed the principal record counts and status distributions. Neither the original production database nor the research agents had to be restarted.
| Archived database | Projects | Other SQL-confirmed records |
|---|---|---|
institute (legacy) |
46 | 1,391 tasks; 535 claims; 19,394 events |
institute_lab2 (legacy branch) |
21 | 866 tasks; 328 claims; 68,020 events |
institute_v02 (later) |
48 | 42 completed projects; 1,045 artifacts; 555 model-usage entries |
Internal model-usage entries in v0.2 totalled US$56.98066. This is an estimated model ledger, not a supplier invoice or a full accounting of staff, servers, experiments and failed retrieval attempts. Manual grouping suggests many project rows were tests or reruns of the same question; that categorisation is exploratory, not an independently established count of unique studies.
We subsequently recovered the complete written final answers for all 42 projects bearing the completed status. They had not vanished: some large text values were stored in PostgreSQL’s compressed TOAST format, which our first extraction missed. We also retrieved a separate Docker artifact archive and checked registered computational files against archived SHA-256 hashes and byte lengths. Hash agreement verifies that the files match the preserved registry, not that their scientific conclusions are true.
Finding 1. “Completed” did not always mean answered
Two of the 42 completed project records concerned Finland’s imports from Estonia. Their full final answers began UNRESOLVED. One documented a 12-step limit without a verifier-accepted answer; another exhausted attempts to obtain usable material from the Finnish Customs service. Yet the workflow status was completed.
That mismatch is directly observable, but its denominator requires care. Two of 42 is the frequency of this explicitly identified contradiction among completed records, not a reliable estimate of the system’s overall failure rate. These were repeated attempts at a topic, and other unsuccessful answers may not contain such an obvious marker.
The general lesson is architectural: completion should describe the end of a software workflow, while research outcomes need separate states such as resolved, partially resolved, unresolved, failed verification and independently reproduced.
Finding 2. The lab really did calculate, and some results reproduce
Four completed projects examined the World Happiness Report 2026. Across these runs, we recovered and hash-checked 103 of 103 registered computation objects: 62 computation manifests and 41 output files. In fresh Python/pandas/SciPy calculations, we recomputed selected statistics from the saved intermediate CSV files rather than copying the agents’ summary JSON.
| Test | Recomputed result | What it means |
|---|---|---|
| WHR 2026 ranking | 147 ranked entities | 143 UN member states ranked, 50 UN members without a rank, and 4 ranked entities outside the UN-193 denominator in our crosswalk |
| WHR 2025 vs 2026 ranks | Spearman ρ = 0.991345 (n = 147) | Numerically reproducible, but the two three-year averages share 2023 and 2024; this is not independent temporal validation |
| Life evaluation vs affect balance | Spearman ρ = 0.665635 (n = 136) | Reproduces from the archived rank columns; exact tie-aware recalculation from input items gives 0.665105 |
The life-evaluation comparison uses archived Gallup 2025 emotional-health data derived from interviews in 2024 and WHR rankings averaged over 2023–2025. The periods are not identical; the correlation is descriptive, not causal. Moreover, the Gallup interactive has changed over time, so full upstream replication requires a properly versioned copy of the original country-level source.
What this proves: several stored computations were real, internally consistent and repeatable from archived intermediate data. What it does not prove: that every source import was faithful, every published conclusion valid, or the same results could be regenerated independently from the original upstream files without additional source preservation.
Finding 3. Correct arithmetic, unjustified statistical interpretation
The strongest methodological finding arose from a rank-uncertainty calculation. One laboratory branch built “conservative rank ranges” by overlapping confidence intervals around countries’ mean life-evaluation scores. Another preserved the published WHR confidence intervals for countries’ ranks. These are not interchangeable statistical objects. The WHR report explicitly distinguishes confidence intervals for means from confidence intervals for ranks.
We compared the two stored representations across all 147 ranked entities. Their endpoints matched exactly in only 27 cases. More importantly, in 25 of 147 cases (17.0%) the overlap-based range did not fully contain the published rank interval. For China, for example, the heuristic gave 49–76, while the published rank interval in the archived table was 61–80.
Reproducing a number is not the same as validating what the number claims to measure. The overlap heuristic may be a useful descriptive aid, but it has no demonstrated 95% rank-interval coverage. Calling it uniformly “conservative” was unsupported.
This is a bounded finding. It concerns one optional procedure in one research branch; it neither discredits the WHR’s published intervals nor shows that the lab’s other calculations were invalid.
Finding 4. Even the definition of a variable matters
We also tested whether a separate correlation depended on the definition of emotional experience. The original laboratory comparison used a net balance formed from five positive and five negative Gallup experience items. The WHR’s positive and negative affect components each use three items: enjoyment, laughter and learning something interesting on one side; worry, sadness and anger on the other. The WHR explains these components; subtracting the negative aggregate from the positive one was our own analytical choice, not an official WHR wellbeing index.
Recalculating both net balances on the same 136 matched observations, with consistent handling of tied integer item scores, gave Spearman ρ = 0.6651 for the five-plus-five balance and ρ = 0.6269 for the three-plus-three balance. The difference was −0.0382; a paired 5,000-resample bootstrap gave a 95% percentile interval of approximately −0.097 to +0.020. Because the interval includes zero, this does not establish a statistically nonzero difference. It does make the influence of operational definitions inspectable.
The broader lesson is not that one affect measure is necessarily superior. It is that a credible automated research process must say what was measured before deciding what its measured result means.
What else did the internal reviewers catch?
Archived records show agents flagging suspect source retrievals, including a purported scientific PDF that was actually a CAPTCHA page and an attempted Cochrane retrieval yielding an error response rather than review data. These are promising examples of provenance checking. We have not replayed every source-to-final-report path, however, so we cannot claim that every flagged error was corrected everywhere downstream.
Ten deliberately chosen individual scientific assertions were also examined against accessible primary reports and abstracts. Several correctly preserved qualifiers, negative results and distinctions between association and causation. Because this was a purposive, non-blinded sample and earlier rounds used somewhat different evaluation thresholds, its scores are not presented as an estimated laboratory-wide accuracy rate. The finished reports can be assessed more comprehensively in a future study; the present report does not certify all 42 as valid scientific work.
What this audit cannot answer
The study has important limits. The same AI-assisted research process helped conduct this audit and prepare this note; it is not external blinded peer review. The development branches overlap, and no randomly assigned single-model or human-only comparison existed. Published papers were not all checked in full, and complete end-to-end regeneration from the original versioned WHR and Gallup source files has not been demonstrated. Estimated model costs exclude meaningful human and infrastructure inputs. Some derived country-level Gallup material is not included in a public data release while licensing and provenance are reviewed.
No independently validated new scientific discovery was established by this audit. This is not evidence that the laboratory produced none. The evidence supports a more specific conclusion: some numerical calculations and source checks worked, a software-success label sometimes concealed an unresolved task, and at least one uncertainty interpretation exceeded its statistical basis.
What should a better AI research experiment measure?
Our next test should compare three workflows using the same pre-specified tasks, source access, budget caps and blinded external assessment: a single-model baseline, a multi-model verification workflow, and the same multi-model workflow with asynchronous written human oversight. Correct, independently adjudicated substantive findings per unit of total cost would be a reasonable primary outcome. Novelty should be rated separately, using an explicit search protocol rather than an agent’s own confidence.
Secondary measures should include citation fidelity, false-confident statements, detection of corrupted or unavailable sources, successful reruns from source data, latency and human active time. Failed and abandoned tasks must remain in the denominator. Without these controls, more agents, more documents and more completed project cards may simply mean more activity.
Downloadable evidence
This limited public companion package contains the documented aggregate results, redacted examples, rank-overlap exception flags, read-only SQL checks, Python reproduction code, dependencies and SHA-256 file hashes.
Scope: These materials support inspection and limited computational reproduction; they are not an independently complete replication dataset. Original WHR confidence-interval tables and source-restricted Gallup data are not redistributed. See README.md inside the ZIP for the exact requirements and limitations.
Methods, data availability and disclosure
Database verification: SQL inventories and status distributions were checked in PostgreSQL 16 on isolated copies of stopped database snapshots. Computation verification: 103 registered World Happiness Report artifact objects matched saved SHA-256 and byte-length records. We recalculated selected metrics in a fresh Python environment from preserved intermediate tables. The compressed computation export had SHA-256 68cd5bd2cfc01b4a9b5316d0ce05bd1bb09fe601f69231b12aac6d432233cdb2. This identifies the private source package, not a publicly accessible dataset or a cryptographic attestation of scientific truth.
Reproduction scripts, checksum manifests, redacted metrics and internal records are retained by The Null Institute. A complete public dataset is not provided here because some source-level material and project logs require confidentiality, copyright and license review. The numbers above are therefore partially computationally reproducible but not yet fully independently replicable from public primary-source inputs. Data or non-restricted code may be requested through the Institute’s contact page; access is subject to source rights and redaction.
Authorship and AI use: Ilya Nikitin, Ph.D., The Null Institute. AI systems assisted the original laboratory, retrospective extraction, calculations, interpretation, verification and preparation of this report. The author is responsible for the published claims. This is an internal research note, not an independent external replication or peer-reviewed article. No grant funder or outside reviewer is implied.
Primary source documentation
- World Happiness Report 2026. Ranking definitions, three-year averaging and presentation of country uncertainty.
- World Happiness Report data-sharing page. Figure 2.1 source data and public-data availability.
- World Happiness Report 2026, Chapter 2. Positive/negative affect components and methodological caveats.
- World Happiness Report 2025. Prior overlapping three-year life-evaluation window.
- Gallup global emotional-health reporting. Context for the 2025 experience-index source vintage.