Show your work: findings you can verify

A finding you cannot verify is hard to trust

Ask an AI tool to analyze a study and you will have a tidy list of themes, and somewhere on that list will be a line saying people found the headline confusing.

The problem starts in the next meeting, when a stakeholder asks a question. How many people actually said that, and who were they? If the answer is not on the page, you have nothing to point to. You can go back to the raw responses and count by hand, or you can ask everyone to trust the summary, and neither option answers the question in the room where it was asked.

Traceable Findings in User Sage shows the responses behind every theme in a Findings readout. The responses sit on the same card as the theme itself, so a reader can verify the claim right there instead of opening a separate methodology note or chasing a footnote.

A finding should carry its evidence the way a quote carries its source. If you cannot trace a finding back to the responses behind it, what you are holding is an opinion that happens to be well formatted.

Two bars looked clear on paper and failed at a glance

Our first version used two proportion bars for each theme. One bar showed how many responses mentioned the theme, and the other showed the split between human and simulated responses.

That version looked complete when we described it on paper, because it covered both the count and the source split. In practice the two bars were the same height and the same width, stacked one above the other, so at a glance they looked like the same chart drawn twice. You had to stop and read the labels carefully to work out which bar answered which question, because the chart gave accurate numbers in a form that still took extra work to understand.

A readout that is technically correct but slow to read is still falling short, because researchers share these pages with people who will give them ten seconds rather than ten minutes. When a reader needs someone to explain the evidence before they can use it, the design has not done the work it was supposed to do, even if the numbers underneath are precise.

We removed both bars and looked for a single visual that could show the count and the source together. Figure 1 places the earlier pair of bars next to the strip that replaced them, using the same theme counts in both versions.

Figure 1. The earlier design used two separate bars for one theme, one bar for the count of responses that mentioned the theme and another bar for the split between human and simulated responses. The strip on the right replaced both bars and shows the same numbers in a single row of slots.

One strip, one slot per response

The replacement is a single strip with one slot for every response in the study. If seven people responded, there are seven slots, and the slots for responses that mention the theme are filled while responses that do not mention the theme leave their slots muted and empty.

Green marks a response from a human participant and indigo marks a response from Simulated people on the AI Panel, so color shows where each filled response came from. Below the strip, a line of text gives the same information in plain words and says that 4 of 7 responses mention this theme, with 2 human responses and 2 simulated responses.

The theme card in Figure 2 shows the strip at the size a reader sees on a Findings page, with the count line sitting directly below it.

Figure 2. A theme card shows one slot for each of the seven responses in the study. Two slots are green for human responses and two are indigo for simulated responses, and the divider labeled 4/7 marks the boundary of the four responses that mention the theme.

The strip shows the size of the study because seven slots stand for seven responses, and a reader can see that four of those slots are filled for this theme. The colors inside the filled slots show that two responses came from humans and two came from Simulated people, so the reader does not need a second chart to trace where the responses came from. Because every response has a slot, the total stays visible while the reader looks at the filled part, and four filled slots out of seven reads very differently from four filled slots out of forty. Keeping that difference visible helps a reader judge how much weight a theme can carry.

We named the component EvidenceStrip and put it in our readout library. Findings themes use it now, and moderated session themes can use it later, so any page that makes a claim from responses does not need a new chart each time.

The strip does reach a limit as the number of responses grows, because each box gets narrower as more boxes are added to the same card width. Drawing one box per response works while each box is still wide enough to see, at roughly eight pixels or more. Past that point, somewhere around forty to fifty responses on a desktop card and fewer on a phone, the strip becomes a continuous bar with the same colors and the same divider. At one hundred responses the individual slots go away, but the reader still gets the exact count from the text line below the bar, and that text line stays in place no matter how many responses the bar has to show. Figure 3 at the end of this section shows both forms with their counts.

Figure 3. The strip keeps individual boxes while each box stays wide enough to see, as the seven response example at the top shows. At one hundred responses the strip becomes a continuous bar, and the divider and the count line still give the exact total of 60 of 100.

A split is not a contest

The strip had one more problem, and we only noticed it once we watched how people read the two colors sitting next to each other. When green and indigo sit side by side inside a filled strip, people tend to read them as two opposing sides, as if human and simulated responses were competing and one side had won. The numbers do not mean that, because the filled slots add up to one total of responses that mention the theme, and the colors only show who is inside that total.

We added a thin vertical line at the boundary between the filled slots and the empty ones, with the fraction 4/7 sitting on the line. Adding that line changed how the strip reads because everything to the left of it now forms one group, the evidence for the theme. Responses to the right of the line sit empty because they did not mention the theme, and inside the filled part the colors still show the mix of human and simulated responses without pulling that group apart.

The header filter follows from the divider as well, because filtering changes the total a reader is looking at and the page has to show that new total plainly. It sits where the participant count already sits, at the top right of the header, and Figure 4 shows the control with the study totals in it. When a reader narrows the view to human responses or to simulated responses, every theme on the page updates to match, including its strip and its denominator, so two of two human responses never gets mistaken for the whole study. Quotes keep their Human and Simulated tags even in a filtered view, because a copied quote should carry its source with it wherever it ends up.

Figure 4. The header control at the top right of a Brief shows the study totals and works as the filter for the page. A reader can view all 7 participants together, or narrow the page to the 2 human participants or the 5 simulated participants.

Simulated responses are data too

While we worked on the chart, we also changed the word we had been using for the human side. Early versions labeled one side “real.” The other side was the AI Panel, so “real” seemed like a convenient shorthand for human.

Simulated responses are data in the same sense as the human responses next to them. Simulated people produced them under defined conditions, and the study stores, counts, and analyzes them the same way it handles any other response. Calling the human side “real” quietly tells the reader that the simulated side is fake, and that implication is wrong. What matters for a reader verifying a finding is provenance, meaning who gave a response, so counts and tags now use the pair human and simulated.

The green used for human responses has meant human from the start, and the visible language now matches that meaning.

Panel names still appear on participant labels, where they identify which panel a person came from. A participant label might say Your Panel, GenPop, or AI Panel, depending on where that participant came from, and that label is about identity. Human and simulated work differently because they roll those panels up into two groups, with human combining Your Panel and GenPop, and counts and filters use that combined group alongside simulated. Keeping identity on participant labels and provenance on counts and filters removed the confusion we had seen in earlier versions, where the same word had to do both jobs at once.

Four rules we kept

None of this needed a new chart type or a clever encoding, because a small set of rules, held consistently across the readout, did most of the work. The rules below are not specific to User Sage, and a research team working in another tool could apply them to its own findings pages.

1. Show the evidence next to the claim so a reader can trace it without hunting. If a page says four people mentioned a theme, a reader should be able to reach those four responses from that statement and open the exact quotes when they want to verify the wording. In our readout the quotes stay collapsed by default and open on demand, which keeps the summary readable while the full responses remain available on the same card.
2. Never let color carry meaning alone, because a reader who cannot distinguish green from indigo would lose information the chart is supposed to show. In the strip, green and indigo show the source mix, and the same information also appears as text below the strip. A reader who cannot tell the colors apart can still read the counts, and a reader who copies a finding into another document takes the numbers along with it.
3. Put the counts in words anyone can understand when they first see the page. A line that says 2 human responses and 2 simulated responses gives the reader the numbers directly, without asking them to decode a legend or work out what an axis means. When a stakeholder has to learn a chart before they can understand a finding, the chart slows down the conversation the finding was supposed to start.
4. Keep the denominator visible so a reader does not have to guess how large a study was. A proportion without its total can sound larger than it is, because the reader fills in a total on their own. Showing all seven slots with four filled gives a small study its proper scale, and a reader who can see the empty slots for responses that did not mention the theme can judge the finding against the whole set of responses.

We will keep using these rules as the readout grows, including on moderated sessions, where themes come from transcripts instead of survey responses. A reader of a transcript theme should be able to trace a claim back to what a participant actually said, in the same way a survey reader can open the responses behind a theme.