Our coverage, script by script — including where we fail

Updated 7 October 2026. Thai and Arabic given names from Wikidata were added: Thai went from 2.5% to 49% end to end, Arabic from 87% to 96.5%, and the overall figure from 88.6% to 91.1%. The table, the Thai section and the hand-labelled result below are rewritten.

Updated 28 September 2026. The table below is from a fresh run of the same fixture. Since the first version of this post, Hebrew went from nothing to 64% end to end after the Israeli birth register was added, Greek from 57.5% to 77.5%, and the US counts behind the most common names were corrected. Paragraphs that no longer held are rewritten.

Every vendor in this category publishes one number. Ours would be 96.6%, and on its own that number is close to useless — it hides which names it was measured on, how many the system refused to answer, and whether the failures cluster in one writing system.

So here is the whole table instead, including the scripts where we are weakest.

What was measured

The fixture is international_accuracy.csv: 4,610 given names across 25 countries and nine writing systems, generated deterministically from per-country male and female name lists. It is not a population sample — nobody is claiming these are the 4,610 most common names on earth. Its purpose is narrower and more useful: a fixed input that anyone can rerun against any version of our engine, or against a competitor, and get the same names every time.

php artisan lookup:fixture
php artisan lookup:benchmark tests/Fixtures/international_accuracy.csv --allow-fuzzy

Three numbers matter, and conflating them is the most common way this metric gets abused:

• Coverage — of all names submitted, how many got any answer at all
• Accuracy when answered — of the names we did answer, how many were right
• End-to-end — of all names submitted, how many got a correct answer

A vendor quoting only accuracy-when-answered can hit 99% by refusing everything difficult.

The table

Kana is in the fixture but only 3 names landed there, which is too few to report as a rate.

Reading the failures

Hebrew was at zero when this post was first written. We had no Hebrew name data, so every Hebrew name came back null — the correct answer when there is no evidence. Since 18 September the engine uses the Israeli Central Bureau of Statistics birth register (counts by name and sex, 1949 to 2024), and Hebrew now answers 66% of the fixture names with 97% of those answers right. The remaining third are names the register does not contain in that spelling.

Thai now answers about half the names. Thai was unsupported for a long time because the obvious approach actively produced harm. Transliterating character by character turns กมล (Kamon) into "kml", which then collides with unrelated entries and returns a confidently wrong answer. Wrong-with-confidence is worse than null, because the caller has no signal to distrust it. A bridge that applies real Thai orthographic rules and looks names up in a dedicated skeleton index fixed the collisions, but with only 22 Thai names behind it coverage stayed at 3%.

Since 7 October the engine also holds Thai given names taken from Wikidata person records (CC0). Coverage is now 51%, and 96.1% of the answers are right. Thailand publishes no counted name statistics, so these names carry no birth counts: answers from them are capped at 95% probability and returned with confidence unverified. The gain comes from direct matches on those names. The skeleton index is still built from Thai-country data only, because building it from the global pool turned 8 of 20 test answers wrong when short skeletons collided.

The remaining gap is names that are not in Wikidata. Closing it needs more Thai name data, not a wider index.

Greek at 80.5% coverage is still a data gap, not an algorithm gap. When we do answer, we are right 96.3% of the time. Greece publishes no counted name statistics, so the remaining gap closes only with more name data.

Han and Hangul invert the usual pattern: coverage is total (100% for Han, 100% for Hangul) but accuracy sits at 90.5% and 91.5%. Chinese and Korean given names carry weaker gender signal than European ones — many characters are genuinely used for both — so the ceiling here is lower for everyone, not just us. Treat probability as load-bearing in these markets rather than reading gender alone.

Why publish the bad rows

Two reasons, one principled and one practical.

The principled one: this product's entire proposition is that you can see where a result came from and decide whether to trust it. Every response carries probability, sample_size and source for exactly that purpose. Hiding the aggregate while exposing the per-name evidence would be incoherent.

The practical one: if you process Thai names, you need to know today, from this page, that we are the wrong tool — not three weeks into an integration. That is cheaper for you and cheaper for us.

The smaller, human-labelled set

Alongside the generated fixture there is a hand-labelled set of 94 names across 24 countries, balanced 47/47 by gender. Result on 7 October 2026: 100% coverage and 98.9% accuracy, 98.9% end to end: 93 of 94 right. Before the Wikidata names, four of the eight Thai rows came back unknown; now all eight are answered and seven are right. The one wrong answer is the Thai name วิไล.

That set is a regression threshold, not a product-accuracy claim. It is small enough that a single bad import would move it, which is precisely what a regression threshold is for. A publishable general accuracy figure would need a much larger holdout set, blind-labelled and split out of the source files — we do not have that yet, and we are not going to imply that we do.