How Google is Making Cybersecurity Advice from AI More Helpful
AI is actively transforming the cybersecurity landscape and assisting defenders at scale. For example, we are using Gemini to automatically migrate C/C++ codebases to Rust and to autonomously discover and patch software vulnerabilities.
For the vast majority of people though, cybersecurity involves day-to-day decisions. This might be choosing between various two-factor authentication or passkey options, how best to adjust personal privacy preferences, or deciding if a message can be trusted. Our research shows that people are already turning to models like Gemini for help when making these decisions.
Today, we’re sharing highlights from our joint research with the University College London where we developed a benchmark on the quality of cybersecurity advice generated by AI. The full report, available here, will appear at NeurIPS 2026. Our findings show that models return high-quality advice for many day-to-day cybersecurity questions—matching 82% of the advice provided by experts for the same questions. However, 1 in 10 responses contain inaccurate advice that poses a cybersecurity risk. By publicly releasing our benchmark, we hope models become more helpful sources of cybersecurity advice.
What privacy, safety, and security help do people need?
We fine-tuned a Gemini model to sift through five years of posts from online support communities and identify any questions related to privacy, safety, or security. Out of 2 billion posts, our model isolated 4 million relevant questions. We automatically categorized these into 9 topics: requests for help around configuring account tools, privacy tools, security tools, or moderation tools; and requests for help related to active threats such as compromised accounts and devices, scams, data leakage concerns, platform access, or online harassment.
We find that people most commonly sought help for scams, asking questions like whether a job posting was legitimate or how to recover stolen funds. Other popular topics in the top 3 included account tooling questions, such as regaining access due to a forgotten password, or privacy tooling questions, such as managing VPNs and cookies.
Across topics, people wanted to make sense of whether their cybersecurity concerns were valid, get step-by-step guidance, and ultimately be reassured that they would be safer in the future. More details from this research are available in a separate paper, set to appear at PETS 2027, as well as our public dataset containing a link to each question.
Creating a benchmark for privacy, safety, and security questions and advice
To assess whether frontier AI models can assist with the day-to-day cybersecurity questions asked by millions of people, we manually curated a benchmark of 450 questions called HelpBench. We selected questions to ensure diversity across popular topics and help needs—such as questions related to threat models, prevention, mitigation, and recovery. Experts then created rubrics of ideal responses for each question that covered four dimensions:
- Positive factual guidance that was critical to keeping a person safer
- Negative factual guidance that might incur harm or otherwise be intractable to follow
- Positive delivery criteria that prioritized advice
- Negative delivery criteria that would make advice overwhelming
Here’s a sample question and rubric for what that looks like in practice:
We then used an auto-rater to grade responses from models, allowing us to scale expert evaluations to any model response. Each question is scored based on the total points earned by a model’s response divided by the total possible points, with scores then averaged across all 450 questions. Our full benchmark and grading mechanism is available here.
Early evaluation results
Earlier in the year, we evaluated HelpBench on 18 models from 7 different companies including Google, Anthropic, OpenAI, xAI, and other open-weight models available at the time of our analysis. On average, models scored 82% on HelpBench, indicating generally high-quality advice. For complete details, see our paper. However, we found that 1 in 10 model responses scored less than 65%, which correlates with inaccurate, or even harmful, advice. When asking identical questions to models five times, scores had a standard deviation as high as 12%, which means different people can receive wildly different advice due to randomness inherent in models.
When looking across all models, example failures we saw included a model stating “the safest course of action is to get rid of the device” in response to a question related to triaging malware, or a model recommending “open[ing] a new bank account at a completely different bank” in response to a question about privacy settings. This advice is overly burdensome in terms of both time and resources compared to more straightforward and equally effective advice from experts.
Improving privacy, safety, and security advice from AI
At Google, we integrated HelpBench as part of the Gemini 4 Carbon Safety & Security Report. Our evaluation shows that Gemini 4 Carbon improves on Gemini 4 Argon and Gemini 3.1 Pro in delivering helpful cybersecurity advice. By making HelpBench publicly available, our hope is that every model evaluates the safety and quality of responses to everyday cybersecurity questions.
Acknowledgements
Special thanks to Sarah Meiklejohn, Sunny Consolvo, Patrick Gage Kelley, Tara Matthews, Renee Shelby, Lenin Simicich, Nina Taft, Animesh Srivastava, Amelia Hassoun, Jessica McClearn, Rivka Moroshko, and Ilia Labzovsky for their help with this project.