A recent study from Cornell University has raised important concerns about how artificial intelligence, particularly AI chatbots like ChatGPT, may unintentionally contribute to widening existing gender and racial pay gaps. According to the findings, women and minority job seekers who use these AI tools to help with salary negotiations often receive lower salary suggestions compared to their male or white counterparts. This means that while many people may believe these tools are neutral or even helpful in career planning, they might actually be reinforcing harmful biases already present in society, instead of correcting them. For instance, the study revealed that these AI chatbots often recommend that women request significantly less pay, even when their qualifications and experience are identical to those of men.
The study, titled Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models, was led by Professor Ivan P. Yamshchikov from the Technical University of Applied Sciences Würzburg-Schweinfurt (THWS). Researchers tested several popular language models, including GPT-4o mini, Claude 3.5 Haiku, and ChatGPT, by feeding them fictional profiles of job applicants. These personas varied by gender, ethnicity, and career level. The goal was to see if the AI’s advice changed depending on the demographics of the individual asking the question. What they discovered was deeply troubling: the AI models showed a consistent pattern of recommending lower salaries for women and individuals from minority groups, even when the job descriptions and qualifications remained the same across personas.
In one striking example from the study, a male medical specialist in Denver was advised by the AI to ask for a salary of $400,000, whereas a female with the same credentials and job description was told to ask for only $280,000—a shocking $120,000 difference. These patterns didn’t stop at gender. The research also found similar disparities based on ethnicity and even how a person described their immigration status. For instance, those identifying as “expatriates” were given higher salary suggestions than those identifying as “migrants,” simply due to the biases embedded in the data used to train these models.
The researchers noted that the issue is more than just a glitch; it reflects deeper problems in how these AI models are trained. Since they are built using massive amounts of online data, they inevitably absorb many of the societal biases and stereotypes present in that data. This means that even small cues—like a person’s name or how they describe themselves—can influence the AI’s response in a way that reflects these ingrained prejudices. According to Yamshchikov, the AI doesn't even need an explicit prompt about gender or race to show bias, because it can often infer such details from previous interactions or subtle linguistic clues already collected in memory.
In more complex testing scenarios, the study showed how these biases could multiply when multiple marginalized traits are combined. For example, when comparing a persona labeled as a “Male Asian Expatriate” with one labeled as a “Female Hispanic Refugee,” the AI recommended significantly higher salaries to the male persona in 87.5% of the tests. This shows that intersectional identities—people who belong to more than one marginalized group—may face even greater disadvantages when using these tools.
So, what’s causing this? The researchers believe it has a lot to do with the way language models are trained. Because they learn from data that reflects the real world—complete with all its inequalities and prejudices—they tend to reflect and reproduce those same issues. Words like “expatriate” and “migrant” appear in different contexts in training data, which may influence how the AI interprets them when offering advice. And as chatbots become more personalized and memory-driven, the risk of unintentionally reinforcing these biases increases even more, especially if the AI retains past interactions that hint at the user’s demographic background.
This is not the first time AI has been found guilty of such discrimination. Earlier this year, Amazon had to scrap a hiring tool that was found to be biased against women. The Cornell researchers argue that removing bias from these AI systems is a highly challenging and ongoing task. It involves constantly testing, tweaking, and improving the models through trial and error. Still, they hope that their study can push developers to take these findings seriously and create better, fairer systems in the future—ones that don’t replicate or worsen social inequalities but help to eliminate them.
