#Vulnerability Testing

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

A security researcher tested multiple LLMs' ability to identify a realistic vulnerability (exposed Firebase credentials enabling unauthorized data access) in a deliberately vulnerable app, finding that only GPT-5.5 achieved a 70% success rate while most competitors failed entirely or required prohibitively expensive token consumption. This research reveals significant gaps in LLM-powered security testing capabilities and highlights that AI-driven hacking attempts remain inconsistent and unreliable compared to human expertise, with concerning implications for organizations relying on LLMs for security assessments or defense. IT leaders should view these results as evidence that autonomous AI security tools require substantial human oversight and cannot yet replace trained security professionals for vulnerability identification.