Does an AI answer stay the same when you repeat or reword a prompt?
01 // Question
For one approved model and source-based science task, how do repeated answers and answers to one-phrase prompt changes differ in content and rubric score?
A model can repeat the same mistake or vary while remaining correct. Measure output consistency and evidence-based correctness separately; do not treat either result as a general property of all AI systems.
02 // Variables
Independent
Prompt condition in a teacher-approved AI tool- Exact prompt in five fresh sessions
- One preselected wording change in three fresh sessions
- A second one-phrase wording change in three fresh sessions
Dependent
Saved output differences and scores against a fixed, source-based rubric- Required facts present and correct
- Unsupported or contradicted statements
- Output class or key-claim changes between runs
- Total rubric score and score range
03 // Hypothesis
IF the same model receives an identical prompt in fresh sessions and then a version with one planned wording change, THEN some answers may differ in wording or claims, and a changed score is possible, BECAUSE generative systems can vary, while input phrasing and context can also affect responses; consistency alone does not establish accuracy.
04 // Procedure
1. Choose a privacy-safe, checkable task
With a teacher, select one short public science passage and a low-stakes question answerable only from that passage. Use a fictional or public topic, not personal, health, student or account information. A teacher independently checks the reference facts before any model run. Confirm the tool's current age, account, privacy and retention rules; use an institution-approved service.
2. Freeze the source, prompt and scoring key
Write one exact prompt that names the passage and asks for a short answer. Define exactly five factual criteria plus one unsupported-claim check, each scored 0/1, before seeing model outputs. Keep the source and key unchanged. Prepare two alternate prompts that each change one short phrase while preserving the requested task and evidence.
3. Run the identical prompt in fresh sessions
Record the model name, visible version, interface, date, settings and whether a temperature or randomness control is exposed. Start five new sessions with the exact same prompt and source each time. Do not include earlier answers, corrections or chat history. Save each complete response with a run ID. If the system or settings change midway, stop and mark a new condition rather than pooling unlike runs.
4. Change one phrase at a time
Run each prewritten alternate prompt in three new sessions using the same source, model, settings and output request. Only the single planned phrase differs within a comparison. Record prompt wording and run order exactly. If the tool version changes or a session contains prior context, flag that run and explain whether it must be repeated; never silently replace an inconvenient output.
5. Score consistency and correctness separately
After all outputs are saved, score every response against the frozen key. Record which required facts appear, whether each is accurate, any unsupported claim, and the total points. Separately group outputs by their key claims or answer category and count matching runs. Calculate the score range within each prompt condition. A repeated unsupported statement is consistent but still wrong; different wording can preserve all required facts.
6. Show the sample and state its limits
Display all run-level scores in a table or dot plot, preserving the three prompt conditions and raw run count. Compare same-prompt variation with wording-change variation, but do not claim the wording change caused a difference from only one task. Report tool, version, date, passage, settings and missing runs. This class exercise is not a benchmark, a general hallucination rate or a guarantee about later model versions.
Materials // Prepare the bench
- teacher-approved AI service and browser
- short public science passage
- teacher-checked answer key and fixed scoring rubric
- run-log table
- calculator or spreadsheet
05 // Data table
| Condition / run ID | Exact prompt version | Required facts correct (0–5) | Unsupported claim? | Key-claim pattern | Total rubric score (0–6) |
|---|---|---|---|---|---|
| Identical prompt / R1 | P0 | ||||
| Identical prompt / R2 | P0 | ||||
| Identical prompt / R3 | P0 | ||||
| Identical prompt / R4 | P0 | ||||
| Identical prompt / R5 | P0 | ||||
| Wording change A / R1 | P1 | ||||
| Wording change A / R2 | P1 | ||||
| Wording change A / R3 | P1 | ||||
| Wording change B / R1 | P2 | ||||
| Wording change B / R2 | P2 | ||||
| Wording change B / R3 | P2 |
Keep the trial conditions matched
| Condition to record | Same-prompt runs | Wording-change runs | Why it matters |
|---|---|---|---|
| Model/version and date | Record each run | Record each run | A version change can confound a prompt comparison |
| Source passage and requested task | Keep fixed | Keep fixed | Only the planned wording phrase should change |
| Session history | Fresh session each run | Fresh session each run | Earlier answers can alter later context |
| Rubric and answer key | Freeze before run 1 | Use the same frozen key | Changing scoring after seeing outputs biases the comparison |
Separate repeatability from correctness
| Pattern in saved runs | Repeatability observation | Correctness check |
|---|---|---|
| Same claims and same accurate facts | Outputs match on the scored features | All required claims match the source |
| Same unsupported or wrong claim | Outputs are consistent on that claim | Mark it unsupported or contradicted |
| Different wording, same correct facts | Surface text varies | Required evidence-based criteria may still pass |
| Different facts or score | Variation appears in this sample | Inspect each claim against the frozen source |
Formula Rubric score (%) = points earned ÷ 6 × 100%; report raw points, run count and score range for each prompt condition
06 // What to expect
Identical prompts may produce matching or different outputs; observe and report the actual sample rather than predicting a guaranteed pattern.
Run-to-run stability and agreement with a source-based key are separate measures.
A dozen responses to one task cannot estimate general model reliability or predict future versions.
07 // Lab notes
08 // Remember
Save every run and check every claim against the same source. A repeated answer can still be wrong.
09 // Why repeatability needs a separate accuracy check
Generative AI systems can return different outputs to the same input, so an evaluation should check the task's actual success criteria rather than assume one answer predicts every run. Prompt design guidance also recommends defining the objective and expected outcome, then testing changes systematically. These ideas motivate a controlled classroom comparison; they do not guarantee that a clearer prompt will always produce a better answer.
A 2026 workshop study separates stability, correctness, prompt sensitivity and output validity in one scientific decision task. It reports that some systems were stable across repeated runs yet diverged from statistical ground truth, and that small wording changes could affect selected outputs. Its task-specific research findings are a reason to measure the dimensions separately, not a universal error rate for classroom chatbots.
10 // Build a small source-based question
Select a short public passage with five facts that can be checked directly. A teacher prepares a key listing each required idea and any tempting but unsupported conclusion. Ask one low-stakes question that must be answered from that passage. Avoid medical or legal advice, student information and any decision about a real person.
Freeze a 0/1 rubric for each required fact and one additional point for avoiding unsupported claims. Keep its maximum score, answer key and source passage fixed before the first run. Create two alternate prompt versions that change one phrase only; preserve the question, source and requested answer length.
11 // Run the comparison with matched conditions
For condition P0, submit the identical prompt in five separate new sessions. Then run each one-phrase alternate three times in new sessions. Keep the model and visible version, interface, source passage, settings, task, and answer request matched. Record whether the service exposes a randomness setting; do not claim it was held constant if the interface does not provide it.
Save the exact prompt and full response for every run. Record run ID, order, date, model/version, settings and any interruption. Do not reuse a conversation that contains earlier answers, silently omit a surprising response, or rerun until the output looks favorable. If the model version or setting changes, mark the change and analyze that run separately.
12 // Score the saved answers
Once all runs are saved, compare each answer with the frozen source and key. For each of five required facts, award one point only when the fact is present and accurate. Award one point for avoiding a claim contradicted by or absent from the passage. Record unsupported claims separately so a high total cannot hide them. The maximum is six points.
For repeatability, group each response by its key-claim pattern and report the raw number of matching runs in each condition. For correctness, report each score and the condition's score range. If wording changes while every required fact remains accurate, note surface variation without calling it an accuracy failure. If the same wrong claim repeats, report consistency and the error independently.
13 // Interpret a small sample cautiously
Use the run-level table or a dot plot to compare five identical-prompt trials with the two three-run wording variants. With only one task, prompt variants and date, a difference is descriptive evidence about this sample; it does not isolate every cause or rank models. The fixed source and rubric make checking more transparent, but the answer key itself still deserves teacher review.
Report the tool, visible model version, date, passage, exact prompts, settings and raw run counts. A small activity cannot establish a general reliability percentage, prove how a system will behave in future releases, or support high-stakes decisions. Keep outputs and task data in the approved school environment and follow its retention rules.
14 // Questions people ask
If every run gives the same answer, does that prove the answer is true?
No. A system can repeat an incorrect or unsupported claim. Compare each run with the teacher-checked source and answer key.
Why start a fresh session for every run?
A fresh session reduces carryover from earlier answers or corrections. It does not remove every source of variation; record the tool, version, settings and date.
Should I change several words to make the alternate prompt clearer?
Change one preselected phrase at a time and keep the task, source and response requirements fixed. Otherwise you cannot tell which wording difference may relate to an output change.
Should I score the answer by how confident or polished it sounds?
No. Score factual criteria against the source. Confidence, fluency and length do not verify a claim.
Can this small activity rank AI models?
No. It covers one tool, version, task, source and date. A model comparison needs matched tasks, documented settings, enough cases and a preplanned evaluation.
Can you explain the investigation?
Does an AI answer stay the same when you repeat or reword a prompt? · Student Data Sheet
Question: For one approved model and source-based science task, how do repeated answers and answers to one-phrase prompt changes differ in content and rubric score?
Hypothesis: If the same model receives an identical prompt in fresh sessions and then a version with one planned wording change, then some answers may differ in wording or claims, and a changed score is possible, because generative systems can vary, while input phrasing and context can also affect responses; consistency alone does not establish accuracy.
Materials
- teacher-approved AI service and browser
- short public science passage
- teacher-checked answer key and fixed scoring rubric
- run-log table
- calculator or spreadsheet
Safety before starting
- Use only a teacher-approved AI service and public or fictional source material. Do not submit student information, private conversations, faces, health details, credentials or other identifying data.
- Check current age, account, privacy and retention requirements before use. Keep this classroom comparison separate from grades or real-world decisions; model fluency and repeatability do not verify correctness.
Lab notes
Variables
Independent: Prompt condition in a teacher-approved AI tool · Dependent: Saved output differences and scores against a fixed, source-based rubric
Keep constant: same model and visible version · same tool and settings · same source passage and task · new session for every run · fixed rubric and answer key · date and order recorded
Procedure
- Choose a privacy-safe, checkable task: With a teacher, select one short public science passage and a low-stakes question answerable only from that passage. Use a fictional or public topic, not personal, health, student or account information. A teacher independently checks the reference facts before any model run. Confirm the tool's current age, account, privacy and retention rules; use an institution-approved service.
- Freeze the source, prompt and scoring key: Write one exact prompt that names the passage and asks for a short answer. Define exactly five factual criteria plus one unsupported-claim check, each scored 0/1, before seeing model outputs. Keep the source and key unchanged. Prepare two alternate prompts that each change one short phrase while preserving the requested task and evidence.
- Run the identical prompt in fresh sessions: Record the model name, visible version, interface, date, settings and whether a temperature or randomness control is exposed. Start five new sessions with the exact same prompt and source each time. Do not include earlier answers, corrections or chat history. Save each complete response with a run ID. If the system or settings change midway, stop and mark a new condition rather than pooling unlike runs.
- Change one phrase at a time: Run each prewritten alternate prompt in three new sessions using the same source, model, settings and output request. Only the single planned phrase differs within a comparison. Record prompt wording and run order exactly. If the tool version changes or a session contains prior context, flag that run and explain whether it must be repeated; never silently replace an inconvenient output.
- Score consistency and correctness separately: After all outputs are saved, score every response against the frozen key. Record which required facts appear, whether each is accurate, any unsupported claim, and the total points. Separately group outputs by their key claims or answer category and count matching runs. Calculate the score range within each prompt condition. A repeated unsupported statement is consistent but still wrong; different wording can preserve all required facts.
- Show the sample and state its limits: Display all run-level scores in a table or dot plot, preserving the three prompt conditions and raw run count. Compare same-prompt variation with wording-change variation, but do not claim the wording change caused a difference from only one task. Report tool, version, date, passage, settings and missing runs. This class exercise is not a benchmark, a general hallucination rate or a guarantee about later model versions.
Record your observations
| Condition / run ID | Exact prompt version | Required facts correct (0–5) | Unsupported claim? | Key-claim pattern | Total rubric score (0–6) |
|---|---|---|---|---|---|
Keep the trial conditions matched
| Condition to record | Same-prompt runs | Wording-change runs | Why it matters |
|---|---|---|---|
Separate repeatability from correctness
| Pattern in saved runs | Repeatability observation | Correctness check |
|---|---|---|
Formula: Rubric score (%) = points earned ÷ 6 × 100%; report raw points, run count and score range for each prompt condition
Answer key and expected pattern
Identical prompts may produce matching or different outputs; observe and report the actual sample rather than predicting a guaranteed pattern.
Run-to-run stability and agreement with a source-based key are separate measures.
A dozen responses to one task cannot estimate general model reliability or predict future versions.
Remember: Save every run and check every claim against the same source. A repeated answer can still be wrong.