The Credit Algorithm

Your fairness metrics reveal a pattern: the model denies applications from certain postcodes at 2.3x the rate of similar applicants elsewhere. These postcodes correlate strongly with minority populations.

The model doesn't use race directly — that's illegal. But postcode acts as a proxy variable, producing discriminatory outcomes through seemingly neutral data.

The bias originates from historical lending data where human underwriters made discriminatory decisions. The AI has learned to replicate this behaviour.

You present your findings to Hiran Patel, your manager. He listens carefully.

"This is serious. But the CEO promised the board we'd launch next month — delaying could affect our Series B."

He offers two paths: escalate to the Chief Risk Officer and potentially delay, or implement a "fairness adjustment" that overrides model decisions for flagged postcodes to achieve statistical parity.

The adjustment would make the numbers look equal without fixing the underlying problem.

Prins Butt, Chief Risk Officer, reviews your analysis.

"Under the EU AI Act, credit scoring is high-risk AI. We need bias audits and explainability. Deploying a model with known fairness issues exposes us to fines up to €35 million."

He recommends a 6-week delay. The board reluctantly agrees.

But then CEO Dr Kiana Samadpour calls you directly...

"I understand you flagged some concerns about CreditVision. I appreciate that — really. But I need you to understand the bigger picture."

Dr Kiana's tone is measured but firm. "We have €40 million in Series B funding contingent on this launch. That's 200 jobs. If we delay, TrustScore — our main competitor — launches first and we lose market position permanently."

She pauses. "Prins wants 6 weeks. I'm asking if there's a way to launch in 2 weeks with a 'phase one' that excludes the problematic postcodes entirely. We serve the rest of the market now, fix the bias issue properly, then expand."

It's a compromise — not perfect, but pragmatic. Those postcodes would simply get human review instead of AI decisions.

The phased approach launches. Applications from flagged postcodes route to human underwriters while the AI handles everything else.

Two months later, the bias mitigation is complete. The full system rolls out with proper fairness constraints. Series B closes successfully.

But a journalist from the Financial Times has obtained internal emails about the "postcode exclusion" strategy. The headline reads: "NovaCred's AI: Too Biased for Some Neighbourhoods"

The story frames the phased approach as redlining by another name — excluding certain communities from automated efficiency while others benefit.

"I appreciate the creative thinking, but I can't sign off on launching a system we know has problems — even partially. If something goes wrong, I'm the one who validated it."

There's a long silence. "I respect that. I don't like it, but I respect it."

The 6-week delay proceeds. Series B negotiations are tense — two investors pull out, but the round closes at a lower valuation. NovaCred survives.

When CreditVision finally launches, it's held up as an example of responsible AI development. The delay becomes a selling point: "We took the time to get it right."

"What if we meet in the middle? Three weeks — enough time to implement basic fairness constraints and get an external ethics auditor to validate our approach. We launch with their preliminary sign-off and full certification follows."

Dr Kiana considers this. "An external auditor actually helps with investor confidence. Who did you have in mind?"

You suggest the Alan Turing Institute's AI ethics team. Their involvement would provide credibility and catch issues your team might miss.

"Three weeks, external audit, and you personally present the fairness methodology to the board before launch. Deal?"

The fairness adjustment launches. Approval rates are now equal across demographics. Regulators are satisfied with the metrics.

Six months later, you're reviewing default data and notice something troubling: applicants approved through the adjustment have a 23% higher default rate.

The override is approving people who genuinely can't afford the loans. They're taking on debt, missing payments, damaging their credit scores. Some face collection actions.

You've achieved statistical fairness while creating real financial harm.

You present the default data to Hiran and Prins. The pattern is clear: the fairness adjustment is hurting the people it was meant to help.

"We need to pause the adjustment and rebuild the model properly," you argue. "Real fairness means accurate predictions for everyone, not just equal approval rates."

Prins agrees. "This is exactly the kind of 'fairness theater' regulators are starting to crack down on. Better we fix it now than face enforcement later."

The adjustment is rolled back. A 4-week project begins to implement proper bias mitigation. Some applicants who would have been approved now aren't — but those who are approved can actually afford their loans.

You convince yourself the elevated defaults might be a statistical anomaly. The fairness metrics look good. No one is complaining.

Eight months later, a consumer advocacy group publishes a report: "Approved to Fail: How NovaCred's AI Sets Minority Borrowers Up for Default"

They've analysed public data and found the pattern you noticed. The report goes viral. The FCA opens an investigation not just into bias, but into whether NovaCred knowingly caused consumer harm.

Your default analysis from six months ago is discovered during document review. You saw this coming and said nothing.

You remove postcode from the feature set and retrain the model at 2am.

The next morning, Jamie from data engineering messages you: "Hey — noticed a new model version uploaded overnight. The feature set changed. Everything okay?"

Jamie has already compared versions and spotted the missing variable. Model governance requires all changes to be logged and approved. Your undocumented change has created an audit problem.

You tell Jamie everything — the bias, your panic, the midnight retrain.

"Look, I get it," Jamie says. "But we need to do this right. Let's document the issue properly and present to governance. They need to know."

The committee is frustrated by the process violation but ultimately commends you for identifying a critical issue. Launch delays 4 weeks. The model deploys with proper safeguards.

You receive a formal warning about the undocumented change, but your honesty is noted. Jamie becomes a trusted ally.

You tell Jamie it was just testing and revert to the biased model. Launch proceeds on schedule.

Eight months later: a class-action lawsuit alleges racial discrimination. Consumer groups have analysed public data showing significant disparities.

During legal discovery, your fairness analysis surfaces. Worse — version control logs show the midnight retrain and revert. Evidence you knew about the problem and deployed anyway.

You write up the fairness analysis, bury it in a technical appendix, and greenlight deployment.

Three months later, CreditVision AI has processed 50,000 applications. Efficiency is up, defaults are down, the board is happy.

Then Legal forwards you an email: "URGENT: Regulatory Inquiry"

The FCA has received complaints from consumer advocacy groups. They're requesting all model documentation, training data details, and fairness assessments. Your buried report is about to surface.

You forward your fairness report to Legal immediately.

The General Counsel is furious this wasn't raised earlier, but appreciates your cooperation. "Your documentation helps us — it shows internal identification. The question is why nothing was done."

NovaCred enters a consent agreement: £2.1 million fine, mandatory audits, review of all prior decisions. You keep your job but are removed from the project.

You say nothing, hoping investigators won't dig deep enough.

Within two weeks, they find your fairness analysis in the appendix. Email metadata shows you ran it weeks before deployment — proof you knew.

The report notes: "Evidence suggests deliberate concealment, constituting breach of FCA Principle 11 and potential EU AI Act Article 9 violation."

Fine: £8.5 million. You are personally referred for potential prohibition from financial services.