How accurate is gender prediction from a name? Measured by script and by probability

Gender prediction from a first name is right about 96% of the time when it gives an answer, and it gives an answer for about 88% of names. That is what we measured on 4,610 labelled names from 25 countries: 3,890 were answered correctly, 84% of the whole set. The spread behind that average is wide. Latin-script names were answered correctly 95% of the time end to end; Greek 58%; Thai and Hebrew, which we do not support yet, almost never.

Those three numbers are the ones to ask any vendor for, including us:

A single "accuracy" figure without coverage next to it can mean almost anything. A service that answers only the easy half of a list can report 99%. How to compare gender APIs goes through this in detail.

Accuracy by writing system

The biggest single factor is the script a name is written in, because it decides whether a name can be matched directly, needs transliteration, or cannot be read at all.

Read the Greek row carefully: when it answers, it is right 97% of the time, but it answers only 60% of Greek names. That is a coverage problem, not an accuracy problem, and the fix is different. The script-level detail, including what we plan for Thai and Hebrew, is in our coverage by writing system.

Is the probability field worth trusting?

Yes, in the sense that matters: the stated probability tracks how often the answer is right. We grouped every answered name in the benchmark by the probability the API returned and checked how many were correct.

So a threshold works the way you would hope. If a use case cannot tolerate one wrong answer in eight, drop everything under 90 and treat it as unknown; you keep about 94% of the answers and the error rate falls to under 3%. The guide to probability, confidence and total_names explains how to combine the fields.

A probability of 100 is not a guarantee. It means the counted records for that name are overwhelmingly one gender, and 2.3% of those answers were still marked wrong in the test, partly because of the test itself (see below).

Where the errors come from

We looked at all 720 names that were not answered correctly.

Unsupported scripts: 396 of the 720. Hebrew (200) and Thai (196). These come back as unknown, which is the honest answer, but they still count as misses end to end.

Names that change gender by country. Jean is male in France and female in the United States; the answer depends on the country you pass. We counted 47 common names that do this in official birth records: the full list.

Genuinely ambiguous names. Robin is 66% female worldwide; Paris is 65% female in US records. No system can be certain about these, and one that returns them at 99% is overstating what it knows.

Transliteration misses. Arabic-script names are matched through a consonant skeleton, and a few come out wrong: أنور (Anwar) was answered as female. These are real errors on our side.

Wrong labels in the test set. The labels come from public name lists in the Faker library, which we did not compile. They are independent, which is the point, but they are not perfect. Faker's US male list includes Dagmar and Skye; the US Social Security records list Dagmar as female in all 112 registrations and Skye in all 533, with no male registrations for either. Those rows count as errors for any vendor that agrees with the registration records, so every absolute score on this list is a little lower than the truth.

Names that are not in the dataset at all

The benchmark above measures names the way customers send them, and most real names are in the dataset in some form. A separate test measures the opposite case: we removed 2,000 names from the dataset entirely and asked for them anyway. The API answered only 9.6% of them, all by closest spelling, and 92.7% of those answers were correct. The other 90% came back as unknown.

That is the behaviour we want: when there is no evidence, say so. But it is also why that 92.7% figure should never be quoted on its own: it describes 192 answers out of 2,000 names, not 2,000 answers.

How to measure it on your own data

Averages across 25 countries tell you little about your list. The reliable test is small and takes an afternoon:

1. Take 200 to 500 rows from your own data where you already know the answer.
2. Send them with the country you would send in production.
3. Count coverage, accuracy when answered, and end to end separately, and slice by country.
4. Look at the name field of every wrong answer. If the name that was looked up is not the given name you expected, the problem is extraction, not classification.

The numbers on this page come from php artisan lookup:benchmark (the 4,610-name benchmark, published at NameGender Research) and php artisan lookup:accuracy (the 2,000-name holdout test), with the method in how we measure. They were measured on dataset version 2026.08; when the dataset changes, they are measured again.