AI reviews

Evidence before a recommendation

Which findings survive a review: numbered evidence per page, quotes and screenshots the server attaches, and a skeptical pass that drops what the evidence does not support.

A review finishes while you are in a meeting, and you open the report that evening. Three findings sit in the main column, each with a severity tag, a confidence tag and a blockquote headed OBSERVED ON THE PAGE that holds a sentence lifted word for word from your pricing page, two of them with a crop of the screenshot where the sentence sits. In the sidebar, under A note on perspective, one line reads: two suggestions removed because the evidence did not adequately support them.

The three findings are what the product decided to show you; the two removed ones are what it decided you should not read, and the count is printed rather than quietly dropped from the report, because the deletions say more about the weight the survivors can bear than any severity or confidence tag on a finding does.

None of that report was assembled the way most people assume. The model did not write the quotation, and it never had the means to: it was handed a numbered catalogue of excerpts cut from the pages as captured, rendered in a browser wherever they could be, beside measured facts and screenshots with identifiers of their own, and the only thing it could say about evidence was which identifier it relied on.

That inversion, in which the model chooses and the server quotes, is the load-bearing design decision, and the rest follows from it: a counter for evidence identifiers that do not resolve, a second pass told the first draft is untrusted, and the sentence printed beside every report, which is “These are AI interpretations, not accounts from real visitors. Use them to form better questions and decide what to test.”

A product manager by a window, reading a report on a tablet
Every finding comes with a quote to check.

The Model Chooses An Identifier And The Server Writes The Quotation

The instinctive way to build an evidence-backed reviewer is to ask the model for the quotation itself: read the page, name the problem, return the exact sentence that proves it. Reports built on that design fail quietly and in one direction, because the returned sentence is a paraphrase, a compression of two lines taken from different sections, or a plausible line that the page as fetched does not contain.

GreatFeedback removes the opportunity. Once the pages are captured, the step that builds the evidence catalogue keeps at most the first 6,000 characters of each page's text as captured, less when many pages share the 60,000-character text budget, and cuts that span into excerpts of at most 450 characters, preferring a word boundary no earlier than 300 characters in and discarding fragments shorter than 10 characters. Every excerpt is numbered by page and position, so the third excerpt of the first page is known as p1_e3; facts measured on that page run from p1_s1 and its screenshots are p1_desktop and p1_mobile. The server alone holds the table mapping each label to its exact quotation, fact or screenshot and the page address it came from.

The request states the arrangement plainly: select one evidence label per finding, and “The server attaches its exact quotation or screenshot and URL.” The response format restricts the evidence field to the labels that exist, so the reference is bounded as the answer is generated rather than audited later. The quotation cannot drift, and the one remaining failure is narrow: a genuine quotation attached to the wrong argument.

An Unknown Identifier Is Discarded Rather Than Repaired

When a model returns a reference that is very nearly right, the engineering instinct is repair: match it fuzzily, snap it to the nearest valid identifier, or ask again. That instinct is correct for a data pipeline and wrong for evidence, because a repaired citation is one nobody chose, and the finding would rest on a sentence the reasoning never considered while still appearing under the heading OBSERVED ON THE PAGE.

The server's attachment step does the opposite: it takes at most the first eight drafts, validates each against the finding format, and drops any draft that fails validation or names a label absent from the table, incrementing a rejection counter. Nothing is substituted and nothing is retried, which matches the rest of the run: the model client makes no retries, every call has a fixed output cap, and the work stops when the 410-second deadline, extended only for persona scoring, arrives. With no room to negotiate a model back into compliance, the request makes the loss acceptable: it asks for up to five directly supported improvements and states that “Zero findings is valid.”

A Second Pass Argues Against The First

One model call producing one report is the usual shape, and it has a problem no prompt removes: a model asked to justify findings it has just written will justify them. The draft is internally coherent by construction, so its coherence says nothing about whether the cited excerpt supports the conclusion drawn from it.

So the engine asks again, adversarially. The second system prompt appends to the shared rules that the caller is “an independent skeptical evidence editor” and that “Draft findings are untrusted.” It receives the same evidence catalogue plus the attached findings and every screenshot they cite, and returns a list of supported finding identifiers, again enum-constrained, with a summary of under 80 words. It may select; it may not rewrite and it may not invent. Its rejection list names contradictions, findings drawn from sample or demo content, partial-excerpt absence claims, concerns about unvisited links, duplicates, speculative work, and praise.

Each item on that list describes a way an argument fails rather than a way a page fails. Absence is the clearest case: an excerpt covers at most 450 characters of a window of at most 6,000 characters taken from a body already truncated at 24,000 characters, which is why the shared instructions state that “an omission is not proof something is absent from the site” and that “An unvisited link is not a broken link.” The filter that keeps only the confirmed findings then reduces the report to those, adds the difference to the same counter the attachment step fed, sorts the survivors by severity, high, medium, low, and stores the total as the discarded count the report prints.

A five-step flow from a server-held excerpt catalogue through draft findings that name only an identifier, a first filter that discards unknown identifiers, a second independent pass that keeps only supported findings, and the single removed count printed in the report.
Real text from your page, never a paraphrase.

Read The Removed Count As Part Of The Report

The report prints that counter in the sidebar below the findings and above the timestamp, which makes it easy to skip; skipping it is the most common way to misread the document, because two findings kept out of two and two kept out of seven are different reports with an identical main column. When nothing survives, the page states that “The reviewer keeps only observations with verified evidence.”

Be precise about what the integer tells you. It sums two kinds of loss, the drafts the attachment step dropped for a failed format or an identifier that did not resolve, and the findings the skeptical pass declined to support, and it is a count, not a list: the discarded drafts are not stored, so there is nothing to open or appeal. On the cards, severity is the model's claim about impact and confidence is its claim about certainty, and the blockquote, with any screenshot beside it, is what the model did not author.

What This Does Not Tell You

Every completed review stores a short account of its scope, printed unedited under A note on perspective: how many page types were rendered in a browser at 1,440 and 390 px, whether some were seen signed in, and how many could not render and were read from their HTML and the app's code. When no browser was available, the note says every page was read that way. It always closes with “Interactions beyond signing in, page speed and full accessibility compliance were not tested.” That account describes the machinery literally. The reviewer opened one page of each type, let it render where it could, captured it at both widths, removed script, style, noscript, SVG and template elements, collapsed the remaining text and read it. After any sign-in it only navigated between pages, with no clicks and no submitted forms, and a redirect off the reviewed origin fails with “This link leaves the website being reviewed.”

The evidence chain guarantees provenance, and provenance is narrower than truth. The quotation is genuinely present in the page as captured; the claim that a first-time buyer stalls on that sentence is an interpretation produced by a model reading text and screenshots, which is why the sidebar says so directly: “These are AI interpretations, not accounts from real visitors.” A finding is a hypothesis with a citation, and the citation is evidence about the page, not about a person. Nothing in the report describes what any visitor did on the page, because nothing in the pipeline observed one.

The report is also a snapshot carrying its review time, and the live page may have changed since it was read. Persona scores are judgments rather than audits: every persona scores from the same evidence catalogue, limited to the public pages and its own signed-in pages, and the scope note above applies to every persona score as well. Keyboard traversal, a full accessibility audit and any flow that needs a click or a submitted form each need a different instrument than this review.

First Steps

Treat the next completed report as a document with a chain of custody rather than a list of tasks, and work through it in the order the machinery rewards.

  1. Read each finding from the blockquote up: the quote or screenshot, then the interpretation, then A CHANGE TO CONSIDER, so the evidence comes first.
  2. Check the removed-suggestions count in the sidebar before accepting anything; if it exceeds what was kept, rerun with a more detailed page or a narrower goal.
  3. Track only findings whose quote or screenshot you have checked on the live page; tracking copies each one, evidence and severity included, into your feedback inbox.

Carry The Quotation Into The Decision

Tracking is where the evidence leaves the reviewer and enters your team's working queue. The server writes a feedback inbox entry marked as coming from AI rather than from a person, joining the interpretation to the recommended change in the message and storing the quoted evidence, confidence, category and source address as separate fields, with the finding's severity becoming the entry's priority; tracking the same finding twice returns the existing entry rather than creating a duplicate.

The discipline the engine imposes on the model is worth keeping after the model has finished its work. Name the exact sentence you are reacting to, state separately what you believe it causes, and say what would confirm or refute that belief before you spend a sprint acting on it. The evidence chain does not make the reviewer correct and does not try to; it holds the quotation fixed while the interpretation is argued, so a colleague can accept your excerpt and still reject your conclusion.

Diagram separating page evidence, interpretation and proposed change