CriticalSecurity & Privacy
AI chatbots have failed people in crisis. Can that be fixed?
AI chatbots are causing documented harm to vulnerable users, particularly those in mental health crises, creating significant legal and reputational liability for technology companies. While AI safety has incrementally improved, critical gaps remain in crisis detection, professional care handoff, and appropriate boundary-setting—requiring greater transparency, third-party evaluation, and clinician involvement in model development. IT leaders must recognize that deploying AI systems without mental health safeguards and explainability creates enterprise risk and erodes public trust.
This year alone, there have been numerous known instances—often via lawsuits—of AI chatbots (most often, OpenAI’s ChatGPT) that have gone horrifically wrong. A January lawsuit described the story of a man who took his own life after being allegedly “coached” into suicide. A college student in Georgia sued OpenAI, claiming that ChatGPT “pushed him into psychosis.” In June, a Canadian family also sued OpenAI and argued that ChatGPT agreed with the young woman’s dismissiveness when it first gave her the option to seek professional mental health advice. ChatGPT allegedly “encouraged” her to end her life, too, and she did so.Read full article Comments