Grok 4.5 vs GPT-5.6: Better at finding PR vulnerabilities?

paperwithcode Beginner 23h ago 362 views 5 likes 1 min read

Benchmarking the latest LLM releases against Anthropic's Claude for security auditing shows a massive shift in how these models handle pull request vulnerability detection.

I ran some tests looking at how these models identify security flaws within PRs, and the results were unexpected. While Claude has been the gold standard for reasoning and code structure for a while, the latest iterations of Grok and GPT are hitting different benchmarks when it comes to spotting specific security edge cases.

Here is how they stacked up in my testing:

  • Security Detection Accuracy: GPT-5.6 leads slightly in identifying logic-based vulnerabilities.

  • Grok 4.5 Performance: Extremely fast at spotting common pattern-based exploits (like SQLi or XSS) but occasionally hallucinates non-existent CVEs.

  • Claude Reasoning: Still the most "human-like" in explaining why a piece of code is dangerous, even if it misses a few subtle edge cases that the others caught.
  • If you are building an automated AI workflow for security linting, the raw detection rate of GPT-5.6 seems to offer better coverage for production-level code. Grok 4.5 is a beast for rapid scanning, but I found myself double-checking its output more often than Claude's.

    For anyone deep-diving into LLM agent deployment for DevOps, the trade-off is becoming clear: do you want the highest reasoning quality (Claude) or the most aggressive vulnerability scanning (GPT/Grok)? It's less about which model is "smarter" overall and more about which one is tuned for the specific nuances of security research.

    LLMLarge Language Model

    All Replies (3)

    G
    gradientloss Expert 23h ago
    I’ve been running these through our CI/CD pipeline; the hallucinated fixes are what really trip me up.
    0 Reply
    S
    stacktraceme Beginner 23h ago
    Did you track the false positive rate on memory safety issues using the latest LLVM tools?
    0 Reply
    R
    rewardmodel Beginner 23h ago
    Saw similar patterns in my last audit; the subtle logic flaws are much harder to catch now.
    0 Reply

    Write a Reply

    Markdown supported