A review finishes while you are in a meeting, and you open the report that evening. Three findings sit in the main column, each carrying a severity tag, a confidence tag, and a blockquote headed OBSERVED PAGE TEXT that holds a sentence lifted out of your own pricing page. In the sidebar, under the heading A note on perspective, one line reads: two suggestions removed because the evidence did not adequately support them.
The three findings are what the product decided to show you; the two removed ones are what it decided you should not read, and the count is printed rather than quietly dropped, because the deletions say more about the weight the survivors can bear than any confidence label does.
None of that report was assembled the way most people assume. The model did not write the quotation, and it never had the means to: it was handed a numbered catalogue of excerpts cut from the pages that were actually fetched, and the only thing it could say about evidence was which number it relied on.
That inversion, in which the model chooses and the server quotes, is the load-bearing decision, and the rest follows from it: a counter for identifiers that do not resolve, a second pass told the first is untrusted, and the sentence printed beside every report, which is “These are AI interpretations, not accounts from real visitors. Use them to form better questions and decide what to test.”
The Model Chooses An Identifier And The Server Writes The Quotation
The instinctive way to build an evidence-backed reviewer is to ask the model for the quotation itself: read the page, name the problem, return the sentence that proves it. Reports built that way fail in one direction, because the returned sentence is a paraphrase, a compression of two lines from different sections, or a plausible line the page does not contain.
GreatFeedback removes the opportunity. Once the crawl finishes, the step that builds the excerpt catalogue keeps the first 6,000 characters of each page's extracted text and cuts that span into excerpts of at most 450 characters, preferring a word boundary no earlier than 300 characters in and discarding fragments shorter than 10 characters. Every excerpt is numbered by page and position, so the third excerpt of the first page is known as p1_e3, and the server alone holds the table mapping each label to its exact quotation and page address.
The request states the arrangement plainly: select one excerpt label per finding, and “The server attaches its exact quotation and URL.” The response format then restricts the evidence field to exactly the labels that exist, so the reference is bounded as the answer is generated rather than audited afterwards. The quotation cannot drift, and the one remaining failure mode is narrow: a genuine quotation attached to the wrong argument.
An Unknown Identifier Is Discarded Rather Than Repaired
When a model returns a reference that is very nearly right, the engineering instinct is repair: match it fuzzily, snap it to the nearest valid identifier, or ask again. That instinct is correct for a data pipeline and wrong for evidence, because a repaired citation is one nobody chose, and the finding would rest on a sentence the reasoning never considered while still appearing under the heading OBSERVED PAGE TEXT.
The server's attachment step does the opposite: it takes at most the first eight drafts, validates each against the finding format, and drops any draft that fails validation or names a label absent from the table, incrementing a rejection counter and moving on. Nothing is substituted and nothing is retried, which matches the rest of the run: the model client is configured to make no retries, a review may spend five model calls, and the work stops when accumulated output tokens plus the next request's budget would exceed 6,000 or the 230-second deadline arrives. There is no room to negotiate a model back into compliance, so the request makes the loss acceptable: it asks for up to five directly supported improvements and states that “Zero findings is valid.”
A Second Pass Argues Against The First
One model call producing one report is the usual shape, and it has a problem no prompt removes: a model asked to justify findings it has just written will justify them. The draft is internally coherent by construction, so its coherence says nothing about whether the excerpt supports the conclusion drawn from it.
So the engine asks again, adversarially. The second system prompt appends to the shared rules that the caller is “an independent skeptical evidence editor” and that “Draft findings are untrusted.” It receives the same excerpt catalogue plus the attached findings, and returns a list of supported finding identifiers, again enum-constrained, with a summary under 80 words. It may select; it may not rewrite and it may not invent. Its rejection list names contradictions, partial-excerpt absence claims, concerns about unvisited links, duplicates, speculative work, and praise.
Each item on that list describes a way an argument fails rather than a way a page fails. Absence is the clearest case: an excerpt covers at most 450 characters of a 6,000-character window taken from a body already truncated at 24,000 characters, which is why the shared instructions state that “an omission is not proof something is absent from the site” and that “An unvisited link is not a broken link.” The filter that keeps only the confirmed findings then reduces the report to those, adds the difference to the same counter the attachment step fed, sorts the survivors high, medium, low, and stores the total as the discarded count the report prints.
Read The Removed Count As Part Of The Report
The report prints that counter in the sidebar as a plain sentence, below the findings and above the timestamp, which makes it easy to skip; skipping it is the most common way to misread the document, because two findings kept out of two and two kept out of seven are different reports with an identical main column. When nothing survives, the page states that “The reviewer keeps only observations with verified text evidence.”
Be precise about what the integer can tell you. It sums two distinct failures, an identifier that did not resolve and a finding the skeptical pass declined to support, and it is a count rather than a list: the discarded drafts are not stored, so there is nothing to open and nothing to appeal. On the cards, severity is the model's claim about impact and confidence is its claim about its own certainty, while the blockquote is the only element the model did not author.
What This Does Not Tell You
Every completed review stores one sentence about its own scope, and the page prints it unedited: “Public-page content review. Visual layout, live interactions, page speed, accessibility compliance, and logged-in workflows were not tested.” That is a literal description of the machinery. The reviewer fetched at most four pages over HTTP, removed script, style, noscript, SVG and template elements, collapsed the remaining text and read it. There were no screenshots, no clicks, no submitted forms and no authenticated session, and a redirect off the reviewed origin fails with “This link leaves the website being reviewed.”
The evidence chain guarantees provenance, and provenance is narrower than truth. The quotation is genuinely present in the page as fetched; the claim that a first-time buyer stalls on it is an interpretation produced by a language model reading text, which is why the sidebar says so directly: “These are AI interpretations, not accounts from real visitors.” A finding is a hypothesis with a citation, and the citation is evidence about the page, not about a person. Nothing in the report describes what any visitor did, because nothing in the pipeline observed one.
The report is also a snapshot carrying the time it was reviewed, and the live page may have changed since. The four lenses are content lenses rather than audits, which the inclusive content lens states in the product's own words when it asks whether the language is clear, descriptive and usable by a broad audience, then adds “This is not a WCAG audit.” Rendered layout, keyboard traversal, contrast ratios and signed-in workflows need a different instrument.
First Steps
Treat the next completed report as a document with a chain of custody rather than a list of tasks, and work it in the order the machinery rewards.
- Read each finding from its blockquote upward, taking the quoted page text first, then the interpretation above it, then the recommendation under A CHANGE TO CONSIDER, so the evidence sets your expectation before the proposed change does.
- Check the removed-suggestions count in the sidebar before accepting anything, and treat a report that discarded more than it kept as an instruction to run again against a more detailed public page or a narrower goal.
- Track only the findings whose quotation you have re-read on the live page, which copies the title, the interpretation, the recommended change, the quoted evidence, the confidence, the category, the page URL and the severity as priority into your feedback inbox.
Carry The Quotation Into The Decision
Tracking is where the evidence leaves the reviewer and enters the working queue. The server writes an inbox entry marked as coming from AI rather than from a person, joining the interpretation to the recommended change in the message and storing the quoted evidence, confidence, category and source address as separate fields, with severity becoming the entry's priority; tracking the same finding twice returns the existing entry rather than a duplicate.
The discipline the engine imposes on the model is worth keeping after the model has finished. Name the sentence you are reacting to, state separately what you believe it causes, and say what would confirm or refute that belief before you spend a sprint on it. The evidence chain does not make the reviewer correct and does not try to; it holds the quotation fixed while the interpretation is argued, so a colleague can accept your excerpt and reject your conclusion.
