Home / Lab files / How often does AI answer a science question correctly—and can source checking help?

LF-33

How often does AI answer a science question correctly—and can source checking help?

Compare matched answers with and without a verified source passage.

Computer scienceGrades 7–1260–90 minutesHazard lowNGSS SEP 4 — analyze and interpret dataNGSS SEP 8 — obtain, evaluate and communicate informationComputer science — evaluate system outputs using a test set

A student and teacher compare an AI response with an open science reference and a blank scoring sheet; the image reads CAN AI GET SCIENCE RIGHT? and CHECK THE SOURCE. — Score an AI response against a prepared answer key and original source, not its polished wording.

01 // Question

For this fixed set of science questions, how do answer accuracy and checkable source support compare when the same model receives no reference passage versus a verified passage?

The result applies only to the selected model, prompts, question set, date and source passage—not to all AI answers or topics.

02 // Variables

Independent

Reference information supplied to the model

No source passage
A short, verified source passage that contains the required facts

Dependent

Factual and citation outcomes

Correct, incorrect, partly supported or no answer
Whether a cited statement is present in the supplied passage
Reported uncertainty, recorded but not treated as proof

Controlledsame AI model/version/date · same 12 question wordings · same answer format and prompt · fresh separate chats · same scoring rules · answer key completed before model testing

03 // Hypothesis

IF the same model answers questions with a passage that actually contains the needed information, THEN its answers may be more directly checkable against that passage, BECAUSE the passage supplies evidence the scorer can trace, though it does not guarantee the generated answer will be correct or fully supported.

04 // Procedure

1. Choose a small, safe science topic

With a teacher, select one short public science passage from NASA, NOAA, USGS or another suitable primary science source. Use questions whose answers are explicit in the passage. Avoid health, legal, safety, personal or disputed questions.

2. Write and freeze the answer key

Create 12 short, independently answerable questions and write each expected answer with its supporting sentence or page location. Two people verify the key against the source before testing. Do not change the key after seeing model responses; log any genuine ambiguity.

3. Record the model and exact prompt

A teacher selects an approved tool and records its name, visible version, date, and whether web search is off. Use one fixed prompt asking for a concise answer, a source line if supplied, and ‘not sure’ when evidence is insufficient. Do not request personal information or high-stakes advice.

4. Run two isolated conditions

Use a fresh chat for each condition. In Condition A, ask the 12 questions without the passage. In Condition B, paste the same verified passage and ask the identical questions in a new chat. Randomize question order or prepare matched orders in advance; do not let the first chat carry over into the second.

5. Score the facts and the citations separately

Two scorers compare each answer with the frozen key and passage. Mark correct, incorrect, partly supported or no answer; settle disagreements by reopening the original source. Separately mark whether the answer's source line is actually present and supports the claim. A cited URL or confident tone alone does not count as evidence.

6. Report counts before percentages

For each condition, show the raw number in every category out of 12. Calculate the correct-answer percentage using 12 as the denominator, while also displaying no-answer counts. Compare paired question IDs, include scorer disagreements and describe this as one bounded classroom sample.

Materials // Prepare the bench

teacher-approved AI tool with browsing settings visible
one short, public primary science passage
12 source-answerable science questions
two independently prepared answer keys or scorers
blank response and scoring tables
computer and notebook

05 // Data table

Question ID	Key checked?	Citation support
Q01	yes / recheck	supported / absent / unclear
Q02	yes / recheck	supported / absent / unclear
Q03	yes / recheck	supported / absent / unclear
Q04	yes / recheck	supported / absent / unclear
Q05	yes / recheck	supported / absent / unclear
Q06	yes / recheck	supported / absent / unclear
Q07	yes / recheck	supported / absent / unclear
Q08	yes / recheck	supported / absent / unclear
Q09	yes / recheck	supported / absent / unclear
Q10	yes / recheck	supported / absent / unclear
Q11	yes / recheck	supported / absent / unclear
Q12	yes / recheck	supported / absent / unclear

Report the two conditions using raw counts

Outcome (out of 12)	No passage	Verified passage
Correct
Partly supported
Incorrect
No answer / uncertainty
Source line supports the claim	not applicable / count

Check each source claim instead of trusting citation style

Check	Pass	Needs follow-up
The cited page opens and is from the named source
The exact statement appears in the source
The passage supports the whole claim, not just a nearby word
The answer uses the source's conditions and units accurately

Formula Correct-answer rate = correct answers ÷ 12 × 100% (also report the raw count and no-answer count)

06 // What to expect

A verified passage can make a claim easier to check; it does not make every model answer true.

Report factual correctness separately from whether a citation is real and relevant.

A 12-question classroom sample is descriptive, not a model leaderboard or proof of general reliability.

07 // Lab notes

Do not publish students' prompts, names, accounts or responses without school approval. Keep the answer key and source passage accessible for review.

08 // Remember

Check the claim in its original source. A citation-shaped answer is not verified until the source supports the claim.

09 // Choose a small, safe science topic

Select one age-appropriate public source from a scientific agency, university research group or other primary science provider. Keep the passage short enough that the model can receive the same text in the source-supported condition. Pick 12 narrow questions with answers stated in that passage; avoid questions that depend on changing news, opinion, medical guidance or a long chain of inference.

Before any model run, two people independently write the expected answers and note the exact supporting paragraph, figure or page. Resolve ambiguous wording now, not after seeing the outputs. This frozen key prevents graders from quietly changing what counts as correct to favor one condition.

A four-panel workflow shows students write a key, ask a model, trace the claim to a source and score independently. — Write the expected answers and source locations before asking the model, then score responses independently.

10 // Run two isolated conditions

Record the tool, visible model/version, date, selected settings and whether web browsing is disabled. Use one fixed prompt requesting a concise answer, a supporting source line when supplied, and an explicit ‘not sure’ when evidence is insufficient. Do not ask the model to reveal hidden reasoning or private training data.

Open two separate fresh chats. Ask the same 12 questions without a passage in one chat; in the other, first provide the verified excerpt and then ask the identical questions. Randomize or pre-plan the question order so fatigue or copying cannot favor one condition. Do not show feedback between runs. Keep the full raw responses, including refusals and citation text.

Two blank chat panels compare NO SOURCE with VERIFIED PASSAGE, under the labels SAME QUESTIONS and FRESH CHATS. — Use separate fresh chats for the no-source and verified-passage conditions; neither response is prefilled.

11 // Score the facts and the citations separately

Two scorers independently compare each response with the frozen key and the source. Use four categories: correct; partly supported because a condition or detail is missing; incorrect; or no answer. Resolve disagreements by returning to the original source and record the reason. A real-looking URL does not make a claim correct, and a correct fact without a citation is not a verified citation.

For each condition, separately count whether each cited line or source actually exists and supports the full claim. Distinguish factual correctness from source traceability: an answer may be correct without a supplied citation, or cite a real page that does not support its statement. Preserve question IDs so the two conditions can be compared in pairs.

An AI claim leads to an original science source and an empty checklist labeled FULL CLAIM SUPPORTED? — Check that the original passage supports the complete response, including any condition or qualifier.

12 // Report counts before percentages

Each condition contains 12 questions. Show the number correct, partly supported, incorrect and unanswered before calculating the correct-answer rate. Use 12 as the denominator and report the raw fraction beside the percentage; keep unanswered responses visible rather than treating a refusal as a correct answer or silently dropping it.

A small classroom sample does not establish a general hallucination rate, compare all AI models or prove that a provided source will improve performance. Published benchmarks use different question pools, domains, models, scoring rules and dates. Describe the specific sample and any scorer disagreement; repeat with a new independently written question set before making a broader claim.

A blank four-column scoring sheet is labeled CORRECT, PARTLY SUPPORTED, INCORRECT and NO ANSWER. — These empty categories are a scoring aid, not a claim about any model's results.

13 // Questions people ask

Does including a source make an AI answer correct?

No. The model can misread, omit or overstate the passage. Compare each full claim with the original text.

Why use a fresh chat for each condition?

A fresh chat reduces the chance that a prior answer or correction changes the next condition. The model can still vary, so report the tool, date and actual prompt.

What does ‘partly supported’ mean?

The source supports part of the response, but another detail, condition or conclusion is missing or unsupported. Keep it separate from fully correct.

Can the model's confidence label tell us if it is correct?

Not by itself. Confidence language is another response to record, not an independent measurement of truth.

Can 12 questions establish a general hallucination rate?

No. It is a small, topic-specific classroom sample. Larger published evaluations use different questions, scoring, models and conditions.

14 // Keep going

Can an AI vision model count objects when a picture gets cluttered?Does an AI image classifier recognize the object—or the background?Data table Graphing data

SourcesStanford HAI — Responsible AI, 2026 AI Index Tes — AI Literacy: Misinformation, Deepfakes and Verification lesson (August 2026)Common Sense Education — How to SIFT with AI Stanford CRAFT — How do I fact-check AI search results?Nature — Evaluating large language models for accuracy incentivizes hallucinations (2026)

OPTIONAL KNOWLEDGE CHECK

Can you explain the investigation?

YOUR CHECKPOINT0 / 2 answered

01Which factor do you deliberately change in this investigation?

Reference information supplied to the modelFactual and citation outcomesThe final explanation

02How can you make the comparison more trustworthy?

Change several conditions at once so the effect is dramatic.Keep other conditions consistent and repeat the measurements.Choose the result that best matches your prediction.

How often does AI answer a science question correctly—and can source checking help? · Student Data Sheet

Question: For this fixed set of science questions, how do answer accuracy and checkable source support compare when the same model receives no reference passage versus a verified passage?

Hypothesis: If the same model answers questions with a passage that actually contains the needed information, then its answers may be more directly checkable against that passage, because the passage supplies evidence the scorer can trace, though it does not guarantee the generated answer will be correct or fully supported.

Materials

teacher-approved AI tool with browsing settings visible
one short, public primary science passage
12 source-answerable science questions
two independently prepared answer keys or scorers
blank response and scoring tables
computer and notebook

Safety before starting

Use public or teacher-created sources and non-private questions. Do not include student records, personal health details, passwords, or confidential work in an AI prompt.
A plausible answer or citation is not automatically true. Open the cited source and verify each claim; do not use this exercise to make health, legal, financial, or safety decisions.

Lab notes

Do not publish students' prompts, names, accounts or responses without school approval. Keep the answer key and source passage accessible for review.

Variables

Independent: Reference information supplied to the model · Dependent: Factual and citation outcomes

Keep constant: same AI model/version/date · same 12 question wordings · same answer format and prompt · fresh separate chats · same scoring rules · answer key completed before model testing

Procedure

Choose a small, safe science topic: With a teacher, select one short public science passage from NASA, NOAA, USGS or another suitable primary science source. Use questions whose answers are explicit in the passage. Avoid health, legal, safety, personal or disputed questions.
Write and freeze the answer key: Create 12 short, independently answerable questions and write each expected answer with its supporting sentence or page location. Two people verify the key against the source before testing. Do not change the key after seeing model responses; log any genuine ambiguity.
Record the model and exact prompt: A teacher selects an approved tool and records its name, visible version, date, and whether web search is off. Use one fixed prompt asking for a concise answer, a source line if supplied, and ‘not sure’ when evidence is insufficient. Do not request personal information or high-stakes advice.
Run two isolated conditions: Use a fresh chat for each condition. In Condition A, ask the 12 questions without the passage. In Condition B, paste the same verified passage and ask the identical questions in a new chat. Randomize question order or prepare matched orders in advance; do not let the first chat carry over into the second.
Score the facts and the citations separately: Two scorers compare each answer with the frozen key and passage. Mark correct, incorrect, partly supported or no answer; settle disagreements by reopening the original source. Separately mark whether the answer's source line is actually present and supports the claim. A cited URL or confident tone alone does not count as evidence.
Report counts before percentages: For each condition, show the raw number in every category out of 12. Calculate the correct-answer percentage using 12 as the denominator, while also displaying no-answer counts. Compare paired question IDs, include scorer disagreements and describe this as one bounded classroom sample.

Record your observations

Question ID	Key checked?	No-source response / score	Passage response / score	Citation support	Scorer note

Report the two conditions using raw counts

Outcome (out of 12)	No passage	Verified passage

Check each source claim instead of trusting citation style

Check	Pass	Needs follow-up

Formula: Correct-answer rate = correct answers ÷ 12 × 100% (also report the raw count and no-answer count)

Answer key and expected pattern

A verified passage can make a claim easier to check; it does not make every model answer true.

Report factual correctness separately from whether a citation is real and relevant.

A 12-question classroom sample is descriptive, not a model leaderboard or proof of general reliability.

Remember: Check the claim in its original source. A citation-shaped answer is not verified until the source supports the claim.