← All insights

What we learned evaluating an agent that answers from company documents

What we learned evaluating an agent that answers from company documents

What we learned evaluating an agent that answers from company documents

What we learned evaluating an agent that answers from company documents

Independent questions, blind grading and stage tests for accuracy, completeness, latency and cost.

Independent questions, blind grading and stage tests for accuracy, completeness, latency and cost.

Sai Yandapalli · November 10, 2026

Editorial image accompanying this field note

Company Brain answers questions from a company’s email, shared files and meeting notes, with source quotes for its claims. It is an agent: a single answer usually takes several searches and reads, and the agent decides what to do after each one.


That makes it awkward to evaluate. A public benchmark tells you how the system does on documents unlike yours. A handful of demo questions tells you how it does on the questions you thought to ask. Neither tells you how often it is right on a fixed test set, or which part of the pipeline to fix when it is wrong.


In this post we describe how we evaluate Brain: how we write questions without leaning on the system under test, how we grade, how we read small differences, and what the evaluations turned up along the way. Accuracy on our main benchmark held at about 80% over the period we cover. The evaluations still paid off elsewhere. Median answer time fell by 56% (103 s → 45 s), and on a test of search alone, the right source went missing 92% less often than before (24% → 2%).


Evaluation workflow: write questions, write gold answers, run the agent, grade blind, read the results.


Writing questions from raw sources


The first rule is independence. If you pick questions by looking at what the system retrieves, you build a test it is bound to pass.


We start from a plain inventory of the benchmark’s documents, metadata only. Seeds (the source documents a question is built from) are drawn from that inventory, never from Brain’s index and never from any answer. We extract text with our own tools, so a bug in Brain’s parser cannot be copied into the test.


Three models each do one job. The first reads a raw source and drafts a question. The second writes the gold answer (the reference answer graders compare against) from the raw text alone. Every quote it cites has to appear verbatim in that version of the document. The third answers the question again from the raw source without seeing the gold answer. If its answer and the gold answer disagree, we drop the question. The re-derivation step exists because a verbatim quote shows the evidence is real, while the answer built on it can still be wrong.


The benchmark has seven kinds of question: a single fact, the latest value when versions supersede each other, who said what in an email thread, fetching a named document, a roll-up across documents, and two kinds with no retrievable answer. In “access denied” questions the answer exists but sits somewhere the asker may not read. In “not in the corpus” questions it does not exist at all. We keep an access-denied question only when no document the asker can search contains the value. We keep a not-in-corpus question only when a search of the sources finds nothing that answers it. Each question is asked with a specific set of permissions, because the correct answer depends on who is asking.


Gold answers written this way still had defects. In our first review of drafted questions about four in ten had something wrong with them, and we fixed or dropped each one. We also retired one question whose source no longer supported its gold answer.


Blind grading and where misses happen


Answers go to graders as blind packets: the question, the gold answer and the agent’s answer under a neutral id, with nothing to show which version of the system produced it. We grade in two passes, each shuffled differently. Each verdict is bound to a hash of the exact packet it judged, so a changed question, gold answer or agent answer cannot be scored with a stale verdict.


The graders are models. They run as separate agents that can read their batch and write verdicts and nothing else, because the answers they read are untrusted model output.


We built a blind label sheet for calibrating them against human labels, and on an earlier benchmark re-judging a sample flipped no verdicts. We have not yet calibrated them.


A score alone does not tell you what to fix, so every miss is attributed to a stage. The right source was never retrieved. It was retrieved and ranked too low. The evidence was found and not linked to the right subject. The agent had the evidence and still answered wrongly.


Reading small differences


Successive full runs scored 83% and then 80%. On a held-back set, the subset of questions we use only for scoring revisions and never for writing them, successive measurements gave 80%, 84% and 86%.


A paired comparison (a McNemar test on the questions that flipped between pass and fail) showed no significant difference. The noise floor, the smallest change we could tell apart from run-to-run variation, was about 7 points on the paired difference between two full runs. So we report about 80%, with no change we can detect at this size. The held-back set is a subset of the questions, so its rise is weaker evidence still. We also score each instruction change on that set, so repeated gains on it can overstate the real gain.


Full benchmark accuracy: 83% then 80%. Held-back set: 80%, 84%, 86%. Noise floor about 7 points.


We compare runs question by question instead of by their averages, and we look for confounds before crediting a change. Our full runs used different cloud routes to the same model, so we ran a control on the original route the same night. It gave the same accuracy.


The breakdown by kind of question was more useful than the average. On the latest run the agent did best on access-denied questions and worst at saying who said what in an email thread. Group sizes differ and the smaller groups swing between runs, so we read these as direction. Permission isolation also gets a second, mechanical test: 6,340 permission cases run through search, with no leaks.


Accuracy by question type: access denied 100%; fetch document 91%; not in corpus 84%; single fact 81%; roll-up 79%; latest value 74%; thread attribution 64%.


Testing one stage at a time


End-to-end accuracy does not say which stage failed. For the stages that mattered, we wrote a stage test.


One runs search without the agent. For each question we call search with that question’s permissions, once with the question’s wording and once with the wording of the gold quote, and record where the right source ranks. That separates “search could not find it” from “the agent did not use it”.


On that test, widening the candidate pool (the documents search retrieves before ordering them) helped more than switching to a stronger ranker (a model that reorders the pool by relevance). Against what was then running, the ranking changes cut the share of questions where search missed the right source by 92% (24% → 2%), and the change shipped.


Search experiment comparing candidate pool and ranker changes.


We wrote stage tests for citations and spreadsheets too. On the citation test, missed contradictions between a claim and its source fell by 68% (31% → 10%), and wrong supporting quotes fell by 93% (14% → 1%). Given the right sheet, the agent answered 89% of spreadsheet questions correctly when it read the sheet as text, and 100% when it read typed rows. We rejected a quick fix that scored 79%.


One caution: an early stage test was too easy. The existing code already made zero overclaims (claims its source does not support) on it, so it could not tell two versions apart.


Two more benchmarks use no deployment data at all: a synthetic property portfolio and a public enterprise-search benchmark. They let us compare configurations of memory and search on fixed, shareable questions. Across four configurations, accuracy on the synthetic portfolio rose from 58% to 90%, and on the public benchmark from 23% to 64%.


Stage test measurements for search ranking, synthetic portfolio, public benchmark, two citation checks, and spreadsheets.


Several of these tests are now at or near 100%: search ranking at 98%, the two citation checks at 99% and 100%, and spreadsheets at 100%. That says as much about the tests as about the system. A test the system fully passes can still catch a regression, but it can no longer show an improvement, and it tells us little about the harder questions we have not written yet. We take a score near the ceiling as a sign that the test needs replacing with a harder one, so the next work here is on the evaluations themselves: harder questions, more adversarial cases, and richer synthetic data.


Four objectives at once


Every change is judged on accuracy, completeness, latency and cost together. We keep a change only if none of the four gets worse beyond noise, and a tie on accuracy goes to the cheaper or faster option.


The cost number we watch is cost per correct answer: what we spend for each answer that is right. Cost per answer alone rewards a system for answering quickly and wrongly.


The metric has a limit, which is that it gives a wrong answer zero value whether it was a confident error or a cautious one.


On the same questions, cost per correct answer fell by 22% between the first full run and the latest. Answers got faster at the median and among the slowest 5%.


Median answer time 103 to 45 seconds; median search 43 to 11 seconds; 95th-percentile answer 23 to 4 minutes.


We revised the agent’s instructions and scored each revision on the held-back set, twice, with blind graders. The revisions cut mean latency by about 24% and cost per answer by about 12%, with accuracy unchanged within noise. We reverted two changes that did not help: ranking newer documents first, and extra rules for composing answers. We concluded that further wording changes would gain little.


Measuring also told us what to leave unbuilt. A heavier reasoning step over memory cost several times more per answer and was no more accurate, so it did not ship. When we tested a cheaper model for picking a document’s date, it had higher precision and lower overall accuracy than the incumbent. A single score would have hidden that trade-off.


What the evaluations found that we were not testing for


Most fixes in this period, 84%, began with a problem an evaluation found first. The rest began with a user report.


Our first full run surfaced search errors, which we fixed. Loading the benchmark at scale also exposed throughput limits. Documents written per minute went up about tenfold (14 → 148), a large reorganization job took 70% less time (157 min → 47 min), and a filter on person or company dropped to a fraction of a second.


Running evaluations safely


We run evaluations in a separate region and quota, with per-region guards, so they do not compete with production for model capacity.


Every evaluation command takes a hard spend cap and writes a spend ledger, and the run stops at the cap. We account for evaluation usage separately from the product’s. The evaluation harness cannot change any document it reads. Every answer, tool call and search is stored with timings, so latency percentiles and per-search completeness come from the same run as accuracy.


Summary


  1. Build questions and gold answers from the raw sources, never from the system’s output. Even then, about four in ten of our first drafted questions needed fixing or dropping.

  2. Grade blind, in two passes, and tie each miss to a stage.

  3. Compare runs question by question and check for confounds before crediting a change. Our end-to-end accuracy held at about 80%, inside a noise floor of about 7 points.

  4. Use stage tests to find what end-to-end accuracy hides.

  5. Judge every change on accuracy, completeness, latency and cost together.

Limitations


A short series of full runs is a thin trend. Attribution in email threads is the weakest kind of question. Long, multi-step tasks have no test of their own yet. Several stage tests have reached their ceiling and need harder versions. The graders still need calibrating against human labels.


Our headline number did not move. The evaluations told us which stage to fix and which changes not to build.

Explore all insights →