Gender API Accuracy Methodology
A defensible gender API benchmark uses labelled names that match the target population, keeps unknown responses, and reports three numbers: coverage, accuracy when answered and end-to-end accuracy. One percentage cannot describe both how often an API answers and whether those answers are right.
The three measures
Coverage
answered ÷ submitted
How often the system returns a category rather than unknown.
Accuracy when answered
correct ÷ answered
How often a returned category matches the fixture label. A cautious API can score highly here by declining difficult names.
End-to-end accuracy
correct ÷ submitted
How much of the full input receives a correct category. This makes the cost of unknown answers visible.
Two fixtures answer different questions
International regression fixture. The 4,610-row test spans 25 countries and nine writing-system groups. It measures the complete lookup pipeline, including fuzzy matching, and exposes segment regressions. It is not described as unseen data.
Holdout fixture. The 2,000-name test is built from records excluded from the lookup index. It measures generalisation beyond indexed evidence with AI fallback disabled and a minimum evidence rule of 100.
The current holdout run answered 9.6% of 2,000 names and was correct on 93% of those answers. The low coverage is published because answered-only accuracy without it would be misleading.
Sampling and labelling rules
- Keep the real country and script mix. Do not replace difficult rows with clean Latin-script names.
- Apply the same input and country hint to every run being compared.
- Preserve unknown answers. Do not impute them to the majority category.
- Report sample size beside every segment percentage.
- Keep ambiguous names and malformed inputs when they occur in the target workflow.
- Do not use inferred gender as an individual identity or for consequential decisions.
Reproduce a run
The commands write versioned JSON reports used by the public pages. A changed report therefore updates both the visible tables and their structured data.
php artisan lookup:accuracy --sample=2000 --min-evidence=100
php artisan lookup:benchmark --min-coverage=0 --min-accuracy=0
php artisan testKnown limitations
A fixture label records the category associated with a name in a source population. It does not identify an individual person's gender. Names can be used across genders, change across time and have different distributions by country.
Country samples are not equal in size. Very small groups, especially Kana, should be treated as test coverage rather than a population estimate. The international fixture is useful for regression detection, while your own labelled sample is the relevant purchasing benchmark.
Percentages change when data, normalisation, thresholds or matching logic changes. Every published result therefore carries its measurement date.