We recomputed a popular GEO study from its own numbers

In its first phase it took untranslated websites from Spain and Mexico, written in Spanish only, and counted how often Google AI Overviews cited them for the same questions asked in Spanish and in English. In its second phase it looked at a separate group of 83 Spanish and Mexican sites that already had both a Spanish and an English version.

Here are the four Google AI Overviews comparisons, with the percentage the post states and three ways of turning the two counts into a percentage.

None of the four stated figures comes out of any one formula applied to all four rows. The 22% for translated Spanish sites matches the symmetric difference, but the same formula gives 144, 111 and 50 for the other three rows, not the stated 431, 213 and 59. We could not find a calculation that produces all four, and the post does not show one.

One line in the study's companion hub page cannot be true under any formula: it describes untranslated Spanish sites as receiving 431% fewer English citations. Nothing can fall by more than all of itself. On the printed counts the English figure is about one sixth of the Spanish one.

The headline number, that translated sites gain 327% more visibility in English searches, cannot be checked either way. It compares the translated group with the untranslated group, and the post does not print how many translated sites were in each country, which is what you would need to put the two groups on the same footing.

Read at its printed counts, the study makes a real point for Google AI Overviews: an untranslated Spanish-language site drew about one English-query citation for every six Spanish-query ones. Language match matters there, a lot.

For ChatGPT the same study reports a much smaller gap, 3.5% fewer English citations for Spanish sites and 4.9% for Mexican ones. That is a modest effect, and it sits awkwardly next to the hub page's line that without a translated page you are excluded from citations entirely.

The two phases measured different sites. The untranslated sites were one group, and the translated sites were a separate group of 83 that already had English versions. Sites that chose to publish in two languages may differ in size, age and authority from sites that did not, so the gap between the groups mixes the effect of translation with the effect of being the kind of site that translates. The companion hub page describes the study as translating these websites with the product, which would be a different and stronger design; the post itself describes a comparison group.

The way to measure what translation does is to follow the same sites before and after they add a language, against sites that did not, on the same questions and engines, and to report each share with its sample size and interval. We do not sample non-English questions yet, so we have not done that study either, and we say so on our methodology page.

Most figures in this field travel without the numbers underneath them. Before one goes into a deck or a client report, these checks take a few minutes and catch most problems.

First, find the counts. A percentage with no counts beside it cannot be checked, and a study that prints its counts, as this one does, deserves credit for it even when the arithmetic does not hold.

Second, redo the division. More and fewer are different numbers from the same two counts: 17,094 is 508% more than 2,810, and 2,810 is 84% fewer than 17,094. A stated figure that matches neither is a question for the author.

Third, ask whether the groups being compared are the same sites. A before and after on the same sites says something about a change. A comparison between two different groups of sites says something about the groups, and the change is only one of the ways they differ.

Fourth, check each engine separately. This study found a large effect on Google AI Overviews and a small one on ChatGPT. A single headline figure that blends engines hides the more useful answer, which is where the effect is and where it is not.

Fifth, look for the sample size behind each figure. Ninety-eight sites is a respectable sample for a vendor study. It still produces a range rather than a point, and a figure quoted without its range will be read as more certain than the data allows.