AI for Scientific Literature and Reproducible Research

Build a chain from question to source to analysis, and report exactly how far the evidence reaches.

Two papers appear to disagree. One reports that developers completed a task faster with AI. Another reports that experienced developers took longer. A hurried summary declares the literature inconsistent and averages the percentages.

That shortcut skips the scientific question. Were the participants doing comparable work? Were the tools from the same period? Was the outcome defined in the same way? Did the studies measure the same population and decision?

AI can help organize these questions and extract candidate answers. A defensible research workflow keeps the source material, extraction choices, analysis decisions, and unresolved limitations available for inspection.

Start with a question that determines what belongs in the review

“Does AI work?” is too broad for a useful evidence search. A more specific question is: “How does access to AI assistance affect time to complete defined software development tasks, in the populations and settings studied?”

This wording identifies an intervention, outcome, and task domain while leaving room to examine differences among studies. It does not assume that one average will apply everywhere.

Write inclusion and exclusion criteria before screening. For this example, a relevant study must measure task completion with an explicit comparison condition. A product announcement without measured outcomes might provide context, but it should not occupy the same evidence row as an experiment.

Record the search date, search terms, databases or sites, and screening decisions. AI can suggest synonyms, but you should inspect whether they retrieve the intended research. A few convenient papers are a teaching packet or a narrative selection, not automatically a systematic review.

Record source status and inspection depth

Our public teaching packet contains two research preprints and a later organizational research update. The comparison below uses the inspected arXiv abstracts and metadata for the two papers, plus the inspected text of the update. It is an initial evidence-screening example, not a completed full-text critical appraisal or an exhaustive account of research available in September 2026.

That boundary matters. An abstract may establish the main question and reported result while leaving important analysis details unresolved. “Not checked in the abstract” is different from “not reported anywhere in the paper.”

For a full review, inspect the complete methods, results, supplementary material, and relevant data or code. Check for revisions, corrections, withdrawals, and subsequent publications. A later document may update a result, discuss a new study, or explain a measurement problem; do not assume those are interchangeable.

Build a claim matrix that preserves differences

The first paper’s abstract describes a constrained programming task and reports a 55.8% faster-completion result. Keep that statement tied to the study rather than translating it into a universal workplace productivity rate. The source is Peng and colleagues’ arXiv record.

The second abstract describes 16 developers completing 246 tasks and reports that AI access increased completion time by 19% in its early-2025 setting. It concerns experienced contributors working on mature projects they knew well. The source is Becker and colleagues’ versioned arXiv record.

The later update discusses developers and tasks selecting out of participation, changed compensation, and difficulty attributing time when agents run concurrently. The authors describe the resulting signal as unreliable for measuring the current effect. See METR’s February 2026 research update.

Each row supports a different statement. None licenses “AI always speeds up coding” or “AI always slows down coding.”

Separate demonstrated findings from explanations to test

One possible explanation for different results is that task familiarity changes how assistance helps. Another is that the cost of reviewing generated work differs between a small standalone exercise and an established codebase.

Those are hypotheses to investigate. The comparison table by itself does not isolate their causal effects. Tool generation, participant selection, task characteristics, and study procedures also differ.

Ask AI to maintain two separate outputs: an evidence summary and a hypothesis list. For every hypothesis, identify a study design or additional analysis that could test it.

For example: “Compare assistance effects across pre-specified familiarity groups while measuring comparable outcomes” is a research proposal. It is not a result obtained by rewording the existing abstracts.

This distinction is useful across sciences. A pattern in environmental observations can motivate a mechanism study. A laboratory result can motivate a field trial. The next experiment should be described as proposed until it has actually been performed.

Do not average incompatible percentages

Before combining effect estimates, inspect the outcome definition and statistical scale. A percentage change in completion time is not automatically a percentage change in output per hour. A median, an arithmetic mean, and an adjusted model estimate may describe different quantities.

Consider a separate fictional arithmetic example. If a fixed task takes 100 minutes without assistance and 80 minutes with assistance, time falls by 20%. For identical tasks completed sequentially, the corresponding task rate rises by 25%, because 100 divided by 80 is 1.25. This example illustrates definitions; it does not recalculate either published study.

Also examine uncertainty, repeated observations, missing outcomes, and the target population before any synthesis. If estimates are not sufficiently comparable, a structured narrative comparison may be more defensible than a pooled number.

AI should show the proposed transformation and its assumptions. If it cannot identify the estimator or denominator, it should not manufacture a combined effect.

Write the analysis plan before running the analysis

A reproducible plan should specify the question, input version, unit of observation, inclusion rules, outcome calculation, comparison, and planned outputs. Define how missing data and repeated observations will be handled. Distinguish confirmatory analyses from exploratory ones.

For a teaching reanalysis of an authorized public dataset, the plan might include:

This is a proposed package structure. No published analysis has been reproduced in this manuscript.

Preserve raw inputs and write transformations to separate outputs. A future reviewer should be able to see how a source column became an analysis variable. Avoid editing a spreadsheet manually without recording the transformation.

If code uses randomness, preserve the settings and explain any remaining nondeterminism. If an external model or service is involved, record the available version and configuration information and retain the relevant inputs and outputs under the project’s permissions.

Check code against the scientific question

AI-generated code may run successfully while estimating the wrong quantity. Ask for a plain-language statement of what each analysis computes and compare it with the plan.

Use a tiny synthetic dataset with a known answer to check basic transformations and denominators. Then inspect whether the real data meets the assumptions. A passed arithmetic check does not establish that the statistical design is appropriate.

Look for leakage when predictive models are involved, confounding when making causal claims, and unreported flexibility when many outcomes or subgroups are examined. Multiple analyses can be useful, but selecting only the most attractive result changes how the evidence should be interpreted.

If the output differs from a published figure, investigate the difference. It may arise from a version mismatch, preprocessing, numerical settings, or an error. Do not quietly adjust the code until the picture looks similar and then call the reproduction successful.

Report actions and limitations honestly

Use precise verbs. “Located the repository” means the repository was found. “Inspected the analysis script” means the relevant code was read. “Executed the script” means it ran. “Reproduced the reported result” requires an actual comparison under a stated criterion.

A research assistant should never collapse these stages into “verified the study.” Even a successful computational reproduction does not by itself establish that the original measurements or causal interpretation are correct.

Preserve negative and inconclusive findings. A failed reproduction attempt with a documented environment error is still useful information. A study that does not resolve your question belongs in the report with its limitation, rather than disappearing from the evidence table.

A reusable prompt

Compare these studies by question, population, task, method, outcome definition, reported result, limitations, and relevance. State whether each extraction comes from an abstract, full text, supplement, or inspected data/code. Keep demonstrated findings separate from proposed explanations. Do not invent missing methods, citations, consensus, or a pooled effect. Draft a reproducible analysis plan with input versions, transformations, estimator, uncertainty, environment, outputs, and checks. Clearly distinguish planned work from executed work.

For students: submit the evidence trail

Choose two papers relevant to a course question. Submit the source records, an extraction matrix, one checked claim from each paper, and a short explanation of why their results are or are not directly comparable.

Undergraduates can begin with abstracts and explicitly label the exercise as screening. Advanced students should inspect full methods and supplementary materials appropriate to the assignment. Graduate students can add a preregistered analysis plan or a carefully scoped reproduction attempt.

Follow the course’s rules for AI assistance and disclosure. Keep your notes on accepted and rejected model suggestions. The educational goal is being able to explain the study’s reasoning and limitations without relying on the model to supply them again.

Practice: turn a headline into a research question

Using the two linked preprints, create your own claim matrix. Start with the abstracts, then inspect the full papers if accessible and record which fields your additional reading resolves. Do not fill inaccessible fields from memory or from AI guesses.

Write a 250-word response to: “The studies prove that newer AI is worse for developers.” Identify why the comparison does not isolate tool age as a causal variable. Add a proposed study or analysis that could investigate a narrower version of that claim.

Completion check: Every reported finding has a traceable source and inspection level; the two studies are not averaged without justification; hypotheses are labeled; the analysis plan could be followed by another researcher in principle; and no unperformed reproduction is claimed.

Stretch: Select one figure with authorized public data and code. Record the exact versions, attempt the documented workflow, and compare the output with a predefined criterion. Report success, mismatch, or a blocker with the same precision.