How accurate is the detector itself?
Every share we publish carries a confidence interval. That interval describes sampling error: how much the number would move if we drew the sample again. It says nothing about whether the individual verdicts underneath were correct. Those are two different kinds of error, and most of this category publishes neither. This page publishes the second one.
Precision
100%
95% interval 79.6% to 100%, over 15 of 15
Of the answers we claimed mention a brand, how many really do. Low precision means we would be inflating a customer's visibility, which is the failure this company exists to prevent.
Recall
100%
95% interval 79.6% to 100%, over 15 of 15
Of the answers that really do mention the brand, how many we caught. Low recall means we would be under-reporting, costing the customer credit they earned.
F1 100 across 21 labelled cases (15 true positives, 0 false positives, 6 true negatives, 0 false negatives). Corpus detector-corpus-1.0.0.
Read this number sceptically, and here is exactly why
The corpus is adversarial, not random. Every case in it was written to attack a specific way a matcher can fail: brand names embedded inside longer words, multi-word brands broken across a line wrap, accented characters sitting against a brand, negative mentions that are still mentions. A random sample of real answers would score higher and mean less. Treat this as a floor on field accuracy rather than an estimate of it.
The interval is wide because 15 and 15 cases are a small denominator, and we print it rather than rounding it away. A rate over fifteen cases is not the same claim as a rate over fifteen hundred, and a page that showed you a bare percentage here would be doing the exact thing we criticise elsewhere on this site.
This measures brand-mention detection, the fuzziest and most error-prone part of the pipeline. URL-based citation matching is deterministic and is not what these figures describe.
What it currently gets wrong
No misses on the current corpus.
That is not a claim of perfection, it is a statement about 21 cases. The honest reading is the lower bound above, not the headline. When we find a case it fails, it goes into the corpus permanently and this section fills back in rather than quietly disappearing.
Building this found three real bugs
The first version of the corpus scored a perfect F1, which was not a result. It was evidence the corpus was too easy. Four harder cases were added specifically to attack the implementation rather than its intent, and three of them found defects in code that was already measuring customer sites:
- Only the first occurrence was checked. An answer mentioning a lookalike brand before the real one reported no mention at all. Under-reporting, and invisible without a labelled corpus.
- Multi-word brands died on whitespace. Engines wrap long answers, so a three-word brand split across a newline was never matched.
- The word-boundary test was ASCII-only. Any accented letter next to a brand name read as a boundary, so a word that merely contained the brand counted as a mention. This one over-reported visibility, which is the direction that matters most.
All three are fixed, and all four cases are permanent regression tests. We are publishing the bugs alongside the score because a benchmark that only ever reports good news is marketing with a number attached.
The sampling half of our error reporting is at how we measure, and the rules we hold ourselves to are at methodology. If you find an answer our detector gets wrong, send it to us and it goes in the corpus.