How to compare name-to-gender APIs without trusting anyone's marketing

Every vendor in this category — us included — has an incentive to quote the flattering number. This page is the test we would want a buyer to run on us.

You should not take our word for any of it. Everything below is something you can measure yourself in an afternoon, with a free tier, on your own list of names. That last part matters more than anything else here.

1. Test on your names, not theirs

A benchmark built from US Social Security data will make every vendor look excellent, because that dataset is public, clean, and everyone has ingested it. It tells you nothing about whether the API will work on your customer list.

Take 200 rows from your actual data, label them by hand, and keep the mix honest — if 30% of your customers are Turkish, keep 30% Turkish. Then run that file through each vendor.

This one step invalidates most published comparisons, including any we might be tempted to write.

2. Separate three numbers that vendors merge into one

This is where most of the misdirection lives.

• Coverage — of all names submitted, how many got any answer
• Accuracy when answered — of the answered names, how many were right
• End-to-end — of all names submitted, how many got a correct answer

A vendor quoting only the second number can reach 99% by refusing everything hard. A vendor quoting only the first can reach 100% by guessing on everything. The number that actually predicts your outcome is the third.

Ask for all three, split by country. If a vendor will only give you one, that is itself information.

For reference, our published run across 4,610 names in 25 countries is 87.9% coverage, 96.0% accuracy when answered, 84.4% end-to-end — with two writing systems near zero, which the aggregate would have hidden.

3. Check what happens on an unknown name

Submit deliberate nonsense: Xqzptl, a product SKU, an empty string, a company name like Acme Holdings BV.

You are looking for one thing — does the API say it does not know, or does it invent an answer? A confident wrong answer is materially worse than a null, because nothing downstream can detect it. Every row in your output looks equally valid.

Then check the same on a genuinely ambiguous real name — Andrea (male in Italy, female in Germany and the US), Jean (male in France, female in the US), Noa, Kim. A useful API returns a low probability here, not a coin flip presented as fact.

Two follow-ups worth asking:

• Does it accept a country hint, and does the answer change? If Andrea returns the same result with country=IT and country=DE, the country parameter is decorative.
• Does the response tell you how much evidence it had? A result drawn from 40,000 observations and one drawn from three should not be presented identically. We return total_names and source on every response for this reason.

4. Price the thing you will actually do

Per-request pricing tables are hard to compare because the units differ. Convert everything to cost per 1,000 names at the volume you actually expect, then check three details that dominate the real bill:

Do unused credits expire? This is the single biggest hidden cost in this category, and it is a policy choice rather than a technical one. Read the vendor's own pricing page for the words "per month", "resets", or "expire". If you process a list quarterly, a monthly-expiring plan means paying for twelve months to use four.

Ours do not expire. That is not a technical achievement — it falls out of selling credits rather than subscriptions — but it changes the arithmetic for bursty workloads considerably.

Are duplicates billed? Real customer lists are full of repeated first names. A 200,000-row file typically contains around 16,000 unique names. Ask whether you are billed for rows or for unique lookups, because the difference is an order of magnitude. Our batch processor deduplicates before lookup for exactly this reason.

Are failed lookups billed? If a name returns null, are you charged? At an 88% coverage rate, that is a 12% difference on the invoice.

5. Test the failure modes, not the happy path

Things worth trying before you commit:

• Upload a file with a BOM, semicolon delimiters, and Windows line endings — the three things that break spreadsheet imports in practice
• Send a name with combining diacritics (José composed two different ways in Unicode) and check both forms return the same answer
• Pull the network cable mid-upload and see whether you get a clear error or a half-processed file
• Check what happens to your source file after processing, and whether you can delete results on demand — if you are handling EU personal data, this is not optional

What we are not going to publish

We do not publish head-to-head accuracy tables against named competitors.

We could run one — there is a lookup:compare-genderize command in our codebase for exactly this — but a comparison is only honest if both systems are measured on the same input, on the same day, with the same country hints, and with the sampling method disclosed. Anything less is a number chosen to win. Vendor-run comparisons in this category are almost always the latter, and ours would be no more trustworthy just because we wrote it.

What we will do is publish our own methodology and our own failures, so you can hold the same ruler up to us. The fixture, the commands, and the raw results are in the repository.

If you run this test and we lose on your data, we would genuinely like to see the names — that is how the Thai and Greek gaps in our coverage table got identified in the first place.