Anthropic marketing (and even supposedly technical write ups) sadly has become more hyperbole and less substance over time imo. This technology is so impressive on its own, really feels like shootings themselves in the foot in the long run, but what do I know
Case in point here where they conveniently fail to report the false positive rate, while also saying that if it wasn’t for Address Sanitizer discarding all the false positives this system would have been next to useless
Right now, we accept false positives as long as you can sort them out. I think it's pretty typical that >99% of fuzzer runs don't result in new coverage. Of course they're far from useless without feedback but it's better to have it if you can. I guess the question is does the llm approach have lower costs for validation and triaging vs just fuzzing alone, unclear to me. Anthropic would like people to believe automation is this scary new unknown
Case in point here where they conveniently fail to report the false positive rate, while also saying that if it wasn’t for Address Sanitizer discarding all the false positives this system would have been next to useless