Table of Contents
How Do Biases in Data Shape the AI Systems We Rely On?
Suppose a school awards “Student of the Year” using an AI agent trained on data from the last 50 years of winners. However, for the first 40 years, the school didn’t allow girls to attend. The agent processes the data and concludes that only boys should get the award. Does that make the agent biased?
Yes. But the bias was never in the code. It simply reproduced a blind spot in the data.
This is algorithmic bias. As governments rush to use Artificial Intelligence in public services like healthcare, pensions, and food rations, the stakes are high. It is tempting to think that switching to an AI system will make government decisions fair. After all, computers don’t have personal prejudices, right?
Mask of authority
The truth is that AI models can lend false confidence to blind spots in data. Definitive answers make decision-makers trust an algorithm blindly. Algorithms trained on real-world history can hide human bias behind mathematical authority, automating it at massive scale.
An algorithm on trial
In the United States, judges use software like COMPAS to help decide bail and sentencing. The software predicts how likely a defendant is to reoffend if released.
ProPublica reviewed COMPAS scores of over 7,000 people arrested in Florida and uncovered systemic racial bias. Black defendants who did not reoffend were falsely flagged “high risk” at nearly twice the rate of white defendants, while white defendants who did reoffend were mistakenly labelled “low risk” far more often.
The creators of COMPAS argued that race was never a data point in the algorithm. So how did it end up biased? The algorithm asked questions like: “Have your parents ever been arrested?” or “Do you live in a neighbourhood with high unemployment?” Given decades of segregation, these acted as proxies for race. The software never needed to know race to discriminate by it.
Examples closer to home
The Government of India’s PSB Loans in 59 Minutes platform uses AI to score MSME creditworthiness using GST filings, income tax returns, and bank statements. Many AI models are trained on historical financial data dominated by formal and urban borrowers.
A rural farmer or small-scale artisan may have years of reliable income, but it may exist in an informal economy that the AI is not designed to recognise. AI may read that absence as high risk, locking out rural borrowers and reinforcing the urban-rural divide.
Can the algorithm explain itself?
What happens when an AI model estimates a district’s unemployment rate using mobile data or satellite imagery instead of measuring it directly? Can the system explain that estimate the way it can an actual survey result?
The real test for any government AI system is whether someone can explain exactly where and why it fails.
Four ways to keep the machine honest
- Build for what is Missing: Clean data is not enough. We need infrastructure that fixes fragmentation and captures alternative records, like UPI transactions, cooperative lending histories, and utility payments. Where no data trail exists, let a human decide instead.
- Algorithmic Impact Assessments (AIAs): Just as infrastructure projects need an environmental review, public AI systems should pass an independent bias check before deployment, testing whether they unfairly reject groups like rural women or informal workers.
- Keep a Human in the Loop: When an AI system denies a citizen welfare, health insurance, or a loan, a human should review it, and citizens must have the option to appeal. Wrongful approvals are harder to detect because no one flags a benefit they shouldn’t have received. These need auditing, such as flagging districts whose approval rates diverge sharply from comparable ones. This alone may not prove anything, but it can tell you where to look.
- Teach Data Literacy: Officials don’t need to write code, but they do need to question a machine’s output and feel empowered to challenge it when their dashboard contradicts what they see on the ground.
Bottom line
We know GPS systems are amazing, but we also know they sometimes glitch. It might tell you to drive straight into a lake. But as a responsible driver, you don’t blindly follow the screen. You use your human judgement and override the app. That’s why technology should be our co-pilot, and the human policymaker must always keep their hands on the wheel.
There is no denying that AI has real potential to make public services faster and more accessible. However, as India advances towards an AI-driven future, the ultimate test of fairness in our public systems will not be the brilliance of our code, but the strength of the human safeguards we build around it.