In 2018, Reuters reported that Amazon had quietly scrapped an AI recruiting tool.
The system had been trained on ten years of the company's own hiring. It learned what the company had done. And what the company had done, mostly, was hire men. So the model penalized resumes that included the word "women's," as in women's chess club captain.
Nobody programmed that. The machine just learned the pattern that was already there.
What's the pattern?
Every automated decision system is a mirror of the decisions it was trained on. That's the whole method. It's also the whole risk.
When New York City passed a law in 2021 requiring bias audits for automated hiring tools, it was following a pattern that's now familiar: a new technology inherits an old bias, someone notices, a rule follows. Credit scoring went through it. Lending went through it. Hiring is going through it now.
I've hired for four companies and I know exactly how my own biases work, because I've caught them. The difference with the machine is scale. A biased manager makes a hundred decisions a year. A biased model makes a hundred thousand, in the same direction, with a confidence score.
What should leaders do now?
So, three things.
Audit before you trust. Run the tool on last year's applicants and look at who it would have screened out. If the pattern makes you uncomfortable, you found the bias before a journalist did.
Let it sort, not decide. AI can rank a thousand applications by relevance in a minute. A human should be the one who says no, and be able to say why.
And keep a name on every rejection. The Equal Credit Opportunity Act figured this out for loans in 1974: if you say no, you owe a reason. Hiring is heading there. Get ahead of it.
Amazon's model didn't invent a bias. It found one and made it efficient. Yours will too, unless you look first.
Who at your company can explain the last hundred people your system screened out?
Related: Who is accountable when your bank's AI says no?
- Reuters, "Amazon scraps secret AI recruiting tool that showed bias against women," October 10, 2018.
- New York City Local Law 144 of 2021 (automated employment decision tools), enforced from July 5, 2023.
More like this on LinkedIn. Follow @samradofficial
