Our coverage, script by script — including where we fail

Every vendor in this category publishes one number. Ours would be 96.0%, and on its own that number is close to useless — it hides which names it was measured on, how many the system refused to answer, and whether the failures cluster in one writing system.

So here is the whole table instead, including the two scripts where we currently return almost nothing.

What was measured

The fixture is international_accuracy.csv: 4,610 given names across 25 countries and nine writing systems, generated deterministically from per-country male and female name lists. It is not a population sample — nobody is claiming these are the 4,610 most common names on earth. Its purpose is narrower and more useful: a fixed input that anyone can rerun against any version of our engine, or against a competitor, and get the same names every time.

php artisan lookup:fixture
php artisan lookup:benchmark tests/Fixtures/international_accuracy.csv --allow-fuzzy

Three numbers matter, and conflating them is the most common way this metric gets abused:

• Coverage — of all names submitted, how many got any answer at all
• Accuracy when answered — of the names we did answer, how many were right
• End-to-end — of all names submitted, how many got a correct answer

A vendor quoting only accuracy-when-answered can hit 99% by refusing everything difficult.

The table

Kana is in the fixture but only three names landed there, which is too few to report as a rate.

Reading the failures

Hebrew returns nothing. Not a low score — zero. We have no Hebrew name data and no script bridge for it, so every Hebrew name comes back null. This is the honest state of the product today, and null is the correct answer when you have no evidence. A system that guessed here would produce a 50% coin flip dressed up as a result.

Thai is at 3% and that is deliberate. Thai was unsupported for a long time because the obvious approach actively produced harm. Transliterating character by character turns กมล (Kamon) into "kml", which then collides with unrelated entries and returns a confidently wrong answer. Wrong-with-confidence is worse than null, because the caller has no signal to distrust it.

The current bridge applies real Thai orthographic rules — final-consonant sounds that change by position, vowels written before their consonant, and characters silenced by การันต์ — and looks names up in a dedicated skeleton index. On a 20-pair test, all 20 skeletons match.

So why is coverage still 3%? Because the index is built from Thai country data only, and we have 22 Thai names. We tested building it from the global pool instead: coverage jumps to 20/20, and 8 of those 20 answers are wrong, because short skeletons collide in a 3.7-million-entry pool. Widening the pool buys a coverage number by paying in wrong answers. We would rather report 3%.

The fix is real Thai name data, not a cleverer index.

Greek at 59.5% is a data gap, not an algorithm gap. When we do answer, we are right 96.6% of the time. We simply do not have enough Greek given names.

Han and Hangul invert the usual pattern: coverage is essentially total (99.5%) but accuracy sits at 86.9% and 92.0%. Chinese and Korean given names carry weaker gender signal than European ones — many characters are genuinely used for both — so the ceiling here is lower for everyone, not just us. Treat probability as load-bearing in these markets rather than reading gender alone.

Why publish the bad rows

Two reasons, one principled and one practical.

The principled one: this product's entire proposition is that you can see where a result came from and decide whether to trust it. Every response carries probability, total_names and source for exactly that purpose. Hiding the aggregate while exposing the per-name evidence would be incoherent.

The practical one: if you process Israeli or Thai names, you need to know today, from this page, that we are the wrong tool — not three weeks into an integration. That is cheaper for you and cheaper for us.

The smaller, human-labelled set

Alongside the generated fixture there is a hand-labelled set of 94 names across 24 countries, balanced 47/47 by gender. Current result: 95.7% coverage, 100% accuracy on the 90 names answered, zero wrong answers. All four misses are Thai, for the reason above.

That set is a regression threshold, not a product-accuracy claim. It is small enough that a single bad import would move it, which is precisely what a regression threshold is for. A publishable general accuracy figure would need a much larger holdout set, blind-labelled and split out of the source files — we do not have that yet, and we are not going to imply that we do.