The gap between “we can build this” and “we should build this” is where most AI failures live. Over the past few years working on enterprise ML, I’ve seen technically impressive systems cause real harm — not because of bugs, but because of unchecked assumptions, misaligned incentives, and a lack of structured thinking about consequences.
These are the five principles I try to hold myself accountable to.
1. Understand Who Bears the Cost of Errors
Every model makes mistakes. The ethical question is: who pays for them?
A false positive in a spam filter is an annoyance. A false positive in a credit-denial model can prevent someone from buying a home. A false negative in a medical diagnosis system can cost a life.
Before you optimise a metric, map out the error distribution:
- Who is most likely to be wrongly classified?
- Are errors distributed uniformly, or concentrated in particular demographic groups?
- Is the cost of a false positive vs. false negative symmetric?
In practice: audit your confusion matrix broken down by group. Use fairness metrics — demographic parity, equalised odds, calibration across groups — but don’t treat them as checkboxes. They’re diagnostic tools that surface questions, not answers.
2. Design for Explainability at Deployment Time
You cannot explain a model in production if you didn’t design for it upfront.
SHAP and LIME are useful post-hoc tools. But they’re approximations — they explain the model’s behaviour locally, not the causal story behind a decision. For high-stakes decisions (loan denials, hiring, medical flagging), that’s often insufficient.
Better approaches:
- Use inherently interpretable models where the stakes are high and you can afford the accuracy tradeoff
- For complex models, build a structured explanation layer — key features, their direction of influence, and a confidence estimate — that gets logged alongside every prediction
- Require human review for low-confidence predictions before action is taken
The bar I apply: could a subject-matter expert with no ML background understand why this prediction was made, and challenge it if it seems wrong?
3. Track Distribution Shift as a First-Class Metric
Models trained on historical data assume the future looks like the past. It rarely does indefinitely.
I’ve seen a fraud detection model degrade silently for months because transaction patterns shifted after a major product launch — and nobody had instrumented drift detection. By the time the business noticed, losses had accumulated.
Operationalise drift monitoring:
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=train_df, current_data=production_df)
report.show()
Track feature distributions, prediction distributions, and outcome labels (when available with lag) as continuous signals, not point-in-time audits.
4. Build Contestability Into the System
A system that can’t be challenged can’t be corrected.
Every automated decision that affects a person should have a human-accessible path to contest it. This isn’t just ethically right — it’s increasingly a legal requirement in many jurisdictions (GDPR Article 22, EU AI Act).
Practically:
- Log every prediction with its inputs, model version, and confidence
- Build a review interface that lets domain experts examine and override decisions
- Track override patterns — frequent overrides in a particular segment are a signal your model is underperforming there
Contestability also forces you to make the system legible to non-engineers, which is usually a good thing.
5. Treat “It’s Just a Model” as a Red Flag
When someone says “it’s just a prediction, it doesn’t make the final call” — be sceptical.
Automation bias is real: humans routinely defer to algorithmic recommendations, especially when under time pressure. A model that “supports” a decision often makes that decision in practice. The human in the loop may be nominal.
Design around this reality:
- Calibrate the confidence displayed to humans — overconfident predictions get deferred to more than appropriately-uncertain ones
- Audit decision outcomes when a model recommendation was accepted vs. overridden
- Periodically remove the model from the loop and compare human-only decisions to see whether the model is actually adding value
Closing
These principles don’t resolve the hard cases — they’re thinking tools, not formulas. The hard cases require judgment, stakeholder dialogue, and often willingness to say “we shouldn’t deploy this yet.”
What I’ve found is that raising these questions early, when they’re cheap to answer, is almost always better than discovering the answers in production, when they’re not.
If you’re working on responsible AI deployment and want to compare notes, I’d genuinely like to hear from you.