How Mozilla Leveraged Mythos AI to Detect 271 Firefox Vulnerabilities with Minimal False Positives

When Mozilla’s CTO announced that AI-assisted vulnerability detection meant “zero-days are numbered,” many reacted with skepticism. It sounded like yet another hyped claim—pick a few impressive results, omit the messy details, and let the buzz build. But on Thursday, Mozilla pulled back the curtain on a real-world achievement: using Anthropic’s Mythos AI model to uncover 271 confirmed security flaws in Firefox over just two months—with "almost no false positives." This wasn’t magic; it was a repeatable process. Below, we break down exactly how Mozilla did it, step by step, so you can apply similar techniques in your own security testing.

What You Need

• An AI vulnerability detection model (e.g., Anthropic Mythos or an equivalent with strong code analysis capabilities).
• Access to the target software’s source code (e.g., Firefox, or any large codebase you want to audit).
• A custom analysis harness — Mozilla created one to bridge Mythos and the Firefox codebase. You’ll need to develop or adapt something similar.
• Skilled security engineers to review AI outputs and validate findings.
• Time for iterative tuning — expect to refine both prompts and the harness.

Step-by-Step Process

Step 1: Set Up the AI Model and API Access

Begin by configuring access to your chosen AI model. Mozilla partnered with Anthropic to use Mythos, which is specifically designed for security tasks. Ensure you have the necessary API keys, rate limits, and permissions. Test basic connectivity before moving to code analysis.

Step 2: Build a Custom Harness to Support the Model

This is the secret sauce. Mozilla’s engineers developed a harness that acts as an intermediary between the AI and the source code. The harness structures the input (e.g., breaking code into manageable chunks, adding context) and post-processes the output. It reduces noise and helps the model focus on real vulnerabilities. Without a good harness, you’ll get the “unwanted slop” of hallucinated bug reports that plagued earlier attempts.

Step 3: Feed Source Code into the Harness

Point your harness at the codebase you want to analyze. For Firefox, this meant millions of lines of C++, JavaScript, and other languages. The harness should send code snippets to Mythos in batches, along with metadata like function names, dependencies, and any known issue patterns. This enriches the AI’s understanding.

Step 4: Let the AI Generate Vulnerability Reports

Run the analysis. Mythos will produce a stream of potential vulnerabilities — sometimes hundreds per session. Each report includes a description, code location, and severity assessment. At this stage, many reports might seem plausible, but a large percentage could still be hallucinated. Do not accept them at face value.

Step 5: Apply False-Positive Filtering Techniques

Mozilla found that earlier attempts with other models had high false-positive rates, but with Mythos and their custom harness, the results were remarkably clean. They achieved “almost no false positives.” How? The harness included validation checks (e.g., cross-referencing with existing bug databases, syntax analysis, and heuristic rules). You can replicate this by:

• Building automated sanity checks into your harness.
• Comparing new reports against known vulnerability patterns.
• Requiring multiple AI runs or consensus before elevating a finding.

Step 6: Human Review of Remaining Reports

Even with near-zero false positives, human experts should examine each flagged issue. Mozilla’s security engineers manually triaged the 271 confirmed vulnerabilities. This step catches subtle logic bugs that AI misses and ensures no critical flaw goes unaddressed. Prepare dedicated reviewer time—going through each report takes effort, but the payoff is high confidence.

Step 7: Document and Fix the Vulnerabilities

For each validated vulnerability, file a bug report (e.g., in Bugzilla for Firefox) with details from both the AI and human reviewer. Assign fix ownership, track patches, and re-test. Mozilla’s two-month sprint ended with 271 fixed or patched issues—a significant boost to Firefox’s security posture.

Step 8: Iterate and Improve the System

Treat the first run as a baseline. Analyze which vulnerabilities the AI missed (false negatives) and which prompts confused it. Adjust the harness, refine the model’s context window, and experiment with different code slicing strategies. Each iteration reduces noise and increases yield.

Tips for Success

• Expect initial high false positives — don’t get discouraged. Earlier attempts with other models were “fraught with unwanted slop.” Mozilla’s breakthrough came after refining the harness and model selection. Invest time in tuning.
• The harness is key — without it, you’re asking an AI to analyze raw code without guidance. A good harness structures input and filters output, turning a blunt instrument into a precision tool.
• Combine AI with human expertise — even the best AI will miss context that a security engineer catches. The 271 findings were validated by people, not the machine alone.
• Use in safety-critical code with caution — if your software runs medical devices or flight systems, overreliance on AI could be dangerous. Always verify with traditional methods.
• Keep models and data updated — AI evolves fast. What worked with Mythos today may be obsolete in six months. Run periodic benchmarks to ensure your approach stays effective.

Ready to try this yourself? Start with step two: build that harness. The rest is a matter of iteration — and a healthy dose of human oversight.

Back to What You Need | Back to Steps | Back to Tips