What’s the Worst That Could Happen?
How implementing a premortem approach uncovers potential pitfalls and risks in AI projects
- 16 October 2024
At its core, software testing is about identifying risks and providing information to make informed decisions. The more established a technology, the better we understand the risks, allowing us to tailor our testing approach accordingly.
With newer or transformative technologies, however, the risks become less predictable, making the imagination and critical thinking of the software tester even more crucial. One area where this is particularly true is Artificial Intelligence (AI). While the potential benefits of AI are massive, so are the risks, many of which are still poorly understood.
The Potential Uses and Potential Risks of AI
AI has the power to transform industries and everyday life. Just look at some of its potential applications:
- Autonomous vehicles: Self-driving cars promise to reduce traffic accidents and improve transportation efficiency.
- Chatbots for mental health support: AI-powered tools could provide scalable and accessible mental health care for those in need.
- Health and fitness applications: Personalized fitness plans and real-time health data can help individuals take control of their health.
- Personal finance tools: AI could revolutionize financial management by offering personalized advice and detecting fraud in real time.
- Chatbots for customer service automation: AI-driven chatbots can deliver instant responses to customer inquiries, reducing wait times and operational costs.
- AI-powered image generation and editing: These tools can enhance creativity, allowing anyone to generate professional-grade images with minimal effort.
- HR & recruitment tools: AI can streamline the hiring process, making it easier to match candidates with the right jobs.
But as with any powerful technology, these applications come with significant risks. With those exciting use cases we could potentially see:
- Autonomous vehicles: AI failing to properly recognize obstacles and making poor quality decisions leading to fatal accidents.
- Chatbots for mental health support: Inaccurate advice from AI mental health chatbots could worsen users' conditions.
- Health and fitness applications: Fitness trackers could miscalculate calorie burn, leading users to make unhealthy decisions.
- Personal finance tools: AI-driven credit scoring systems could misclassify users, resulting in unfair loan or reduced credit limits.
- Chatbots for customer service automation: Errors from customer service chatbots could lead to financial losses and damaged reputations.
- AI-powered image generation: AI could produce inappropriate or harmful imagery, including violent or explicit content.
- HR & recruitment tools: AI in recruitment could exhibit bias, excluding qualified candidates, and expose companies to legal risks and damage workplace diversity.
These aren’t hypothetical risks—they’ve already happened! I've included examples of where AI has gone wrong at the end of this blog.
With these sorts of risks, it's important to be cautions as we continue to embrace AI. As software quality practitioners, we need to always ask ourselves: “what’s the worst that could happen?”
One of my colleagues recently emphasized the importance of a "premortem" when adopting transformative technologies.
What is a Premortem?
A premortem is a proactive risk assessment tool designed to uncover potential issues before they become real problems. After a problem has occurred teams will typically conduct a postmortem analyzing what went wrong in hindsight. A premortem turns that process on its head by imagining that the project has already failed before it even begins.
Developed by psychologist Gary Klein, a premortem encourages the team to picture a future scenario in which their project has completely derailed. By asking, “Why did this failure happen?”, team members engage in constructive pessimism, thinking creatively about what could go wrong. This approach taps into the team's collective experience and foresight, allowing them to anticipate risks that might otherwise be overlooked.
A typical premortem process involves:
- Setting the stage: At the start of a project, the team is asked to imagine that the initiative has failed spectacularly.
- Brainstorming reasons for failure: Team members individually write down all the possible reasons why the project could fail.
- Consolidating and discussing: The team comes together to discuss the most critical risks, ranking them based on severity and likelihood.
- Developing mitigation strategies: Once the risks are identified, the team brainstorms ways to address and prevent each potential problem.
The advantage of a premortem is that it sidesteps groupthink, the tendency for teams to avoid raising potential negatives because they’re caught up in optimism. It encourages dissent and creative thinking in a structured environment, allowing for a broader range of risk scenarios to be considered. As technology becomes more complex and transformative—like AI—the risks become harder to predict, making a premortem an invaluable tool for software testing and project planning.
For more on this concept, check out the HBR article on performing a project premortem.
The Critical Importance of Asking “What’s the Worst That Could Happen?”
With transformative technologies like AI, the potential for positive disruption is enormous. However, so too are the risks. AI’s complexity often creates a false sense of security: its sophisticated algorithms can mislead us into thinking they are infallible. But as real-world examples have shown, AI is far from perfect.
When we ask ourselves “What’s the worst that could happen?”, we aren’t being cynical, or anti-progress —we’re being responsible.
This question forces us to confront the less glamorous side of technological innovation: bias, failure, and unintended consequences. It’s only by actively considering these worst-case scenarios that we can add guardrails to protect the things that matter!
Examples of AI Gone Wrong
Autonomous Vehicles
Vehicle Autopilot has been linked to multiple fatal crashes. For example, in 2016, an Autopilot failed to recognize a white truck against a bright sky, leading to a fatal collision. Despite this, similar issues recurred in 2019, sparking lawsuits for not addressing known defects.
Chatbots for Mental Health Support
AI chatbots, while offering instant access to mental health resources, have shown significant limitations. Some have been unable to recognize signs of distress, providing inappropriate or even harmful responses, raising concerns about the safety and efficacy of using AI in mental health care.
https://www.hbs.edu/faculty/Pages/item.aspx?num=64841
Health and Fitness Applications
AI-powered fitness trackers have been criticized for inaccurate readings. For example, studies have shown that calorie tracking can be highly inaccurate, potentially leading to unhealthy decisions by users relying on faulty data.
https://www.wellandgood.com/calorie-counters-on-fitness-trackers/
Personal Finance Tools
Several women reported receiving significantly lower credit limits than men, despite having similar or better financial profiles. This raised concerns about potential gender bias in AI algorithms used for credit decisions. The New York Department of Financial Services (NYDFS) launched an investigation to determine whether these algorithms violated anti-discrimination laws, emphasizing that any algorithm resulting in biased treatment based on gender would be unlawful.
https://citinewsroom.com/2019/11/apples-sexist-credit-card-investigated-by-us-regulator/
Chatbots for Customer Service Automation
AI chatbots have provided inaccurate information, causing financial losses and damaging reputations. For instance, a well-known case involving an airline misinformed customers leading to a legal case and a loss of customer trust.
https://globalnews.ca/news/10297307/air-canada-chatbot-error-bc-ruling/
AI-powered Image Generation
Copilot tools were found to generate inappropriate content, such as violent or sexualized images, raising significant concerns about the ethical implications of AI in creative fields. For instance, users could manipulate prompts to produce unsettling imagery, including violent scenes or over-sexualized depictions.
HR & Recruitment Tools
AI systems used in recruitment have exhibited bias excluding qualified candidates based on factors unrelated to job performance. This not only leads to missed opportunities but can also expose companies to legal risks and diversity issues.