← All insights

How we measure whether Company Brain gets the answer right

How we measure whether Company Brain gets the answer right

How we measure whether Company Brain gets the answer right

How we measure whether Company Brain gets the answer right

How we test answer accuracy, permissions, speed and cost using real company documents.

How we test answer accuracy, permissions, speed and cost using real company documents.

Sai Yandapalli · November 10, 2026

Editorial image accompanying this field note

Ask an AI assistant what rent the bank on your ground floor pays and it will give you a number. What you can’t see is whether that number came from the signed renewal or from a draft someone abandoned two years ago.


PropCo, an anonymized customer, had the problem most owners have. What the company knows sits in years of email, a shared drive with several copies of every lease, and meetings nobody wrote up. Its people were already using Mason’s assistant. It answered questions by searching the mailbox and the shared drive live, one search at a time.


It worked, but slowly. A typical answer from that assistant took about two and a half minutes and cost about four times as much as an answer from Company Brain, which in our latest test answers in about 45 seconds. And nobody, on either side, could say how often the answers were right.


How Company Brain answers a question


Company Brain imports the email, the files and the meeting notes and keeps two things. One is a readable copy of every message, attachment and file. The other is a set of short, dated notes (“the lender approved the third draw on October 2”), each tied to the exact sentence it came from.


When someone asks a question, Brain searches both and hands the assistant the evidence along with where it came from. Every claim in the answer carries its source, so you can see which document it used and check it yourself. If a search comes back incomplete, Brain says so in the answer.


It only shows people what they are allowed to see


Legal and IT teams ask about access first, and they are right to. Permissions follow the systems you already use. A person’s mailbox and meeting notes are readable only by that person. Shared sites are readable by everyone who can already open them.


We test this in two ways. One group of test questions asks for something that exists in the data but sits in a mailbox the person asking cannot read. Brain refused 100% of them in both full runs of the test. A separate automated test runs 6,340 permission cases through search and checks whether anything comes back that the person asking may not read. Nothing did.


How often it is right


Accuracy claims for products like this are usually taken on trust. We would rather show our work, so we built a test from PropCo’s own data.


The questions are written from PropCo’s own files and mail, and each comes with a correct answer copied word for word from the source document. Brain answers every question as a specific person at the company, with that person’s permissions, and each answer is checked against that correct answer.


Accuracy by question type: permission isolation 100%; named-document retrieval 91%; not-in-data 84%; fact lookup 81%; roll-ups 79%; latest value 74%; thread attribution 64%.


Brain answers about four in five questions correctly. We ran the full test twice, nine days apart, and it scored 83% and then 80%. On a separate set of questions we keep aside and never use while improving Brain, the latest version scored 86%. Those scores are close enough that the gaps could be chance.


Brain is strongest where a wrong answer would cost you the most. In testing it never showed anyone something they were not allowed to see, and it found the document you named nine times in ten. When the answer was not in the data, it usually said so.


What happened at PropCo


Weekly questions relative to week one: 1x, 2.5x, 3.8x, and 6.5x.


The team used Brain more every week, and the number of people asking roughly tripled. By the fourth week about 15% of everything the team asked the assistant went through Brain.


We read every one of those questions to see what people wanted. 44% were fact lookups (a tenant’s current rent, a cap on operating costs, who the lender’s lawyer is). Another 34% were larger tasks that used Brain along the way, such as reviewing the mortgage documents for a property or estimating the next construction draw. 17% were requests to find one email or document, and the remaining 5% asked for the status of a project.


People also kept the conversation going.


Follow-up rates: 43% for conversations using Brain and 22% for other conversations.


At the end of the four weeks PropCo agreed to extend Brain to more of the team.


Faster and cheaper over time


We re-run the same test as Brain changes, so speed and cost in testing are measured on the same questions each time.


Answer and search speed improvements: median answer 103 to 45 seconds; search 43 to 11 seconds; slowest 5% of answers 23 to 4 minutes.


The slowest answers are the ones people remember, and they improved the most.


Cost per correct answer fell 22% while accuracy held near 80%.


Cost also fell in daily use at PropCo.


Daily-use typical cost per answer by week relative to week one: 100%, 62%, 27%, 38%.


It keeps getting better


We aim tests at weak spots and measure again after each fix.


Stage test progress for search ranking, synthetic company questions, and citation checks.


Brain is now far more likely to find the signed lease ahead of the drafts that came before it, and the quote beside a claim is almost always the right one.


Most of these fixes began with a problem our own tests found first.


84% of fixes began with a problem our own tests found; 16% were reported by a customer.


Company Brain is live today. See it answer questions from your own documents, with the source beside every answer. Book a demo.

Explore all insights →