That question, asked thousands of times a day across engineering organisations, is where most AppSec programmes quietly fall apart. Every code change carries an implicit decision: does this need a security test? The answer should come from what changed, what it touches, what trust assumptions it makes, and what the business cost of getting it wrong would be. In practice, it comes from intuition, backlog pressure, and whoever happens to be available.

Cytix was built to make that judgement at machine speed and human quality. The core of that capability is a step we call testability evaluation: a structured decision about whether a given change requires security testing, cannot be tested in its current state, or lacks sufficient context to make a confident call. Get this right, and everything downstream improves: the threat model, the test plan, the remediation workflow. Get it wrong, and you have either a backlog full of unnecessary tests or a blind spot where real risk is sitting unexamined.

This piece documents how we approached building that capability, what we found when we benchmarked five large language models against it, and what the results tell us about where LLMs can be trusted with security-critical decisions.

01 The Challenge

Why This Problem Is Harder Than It Looks

Testability classification looks, on the surface, like a contained task. Three labels, a ticket, a clear instruction. In practice it requires contextual reasoning that draws on security domain knowledge, engineering judgement, and an understanding of what constitutes material risk versus noise.

Consider the difference between a change that modifies authentication session logic and one that updates a line of UI copy. The first almost certainly warrants a test. The second almost certainly does not. Between those two obvious cases lives a large grey area: changes to internal APIs, dependency updates, permission model adjustments, logging amendments. These are the tickets where experienced security professionals sometimes reach different conclusions. They are also the tickets where a model either earns its place in the pipeline or becomes a liability.

This is not a problem unique to Cytix. When you apply a basic LLM prompt to a security triage task without domain-specific engineering, a near-even class balance like this one (47% testable, 53% not testable) is no protection against poor results. Naive configurations in our testing scored as low as 17% accuracy: barely better than random, and worse than simply predicting the majority class every time. The AI security market is full of tools making accuracy claims that rest on exactly that kind of baseline. Understanding where the floor is was the starting point for understanding what good actually looks like.

The question we set out to answer was: what does it take to get this right?

02 Methodology

Building the Evaluation

Cytix processes hundreds of changes every day. Our dataset is formed from a sample of these changes, sourced from GitHub pull requests and Jira across actual consenting customer environments. These were not synthetic examples constructed for the purpose. They were the kind of ambiguous, under-documented, context-dependent tickets that AppSec teams encounter every day.

Each ticket was manually labelled by a human security expert using one of three classifications:

• requires_security_testing— the change has characteristics that warrant a security test

• cannot_be_tested— the change is not in a state where meaningful security testing can occur

• not_enough_information— there is insufficient context in the ticket to make a confident call

The manually labelled set became the ground truth against which every model output was scored.

This approach matters for two reasons. First, it grounds the evaluation in the real. We are not testing how well an LLM passes an exam we wrote; we are testing how closely its judgement aligns with the judgement of an experienced security professional looking at the same material. Second, it means the benchmark has an honest ceiling. There are cases in this dataset where even expert humans would disagree. The tickets where no model consistently gets the right answer are, in most cases, edge cases where the correct classification depends on context the ticket does not surface. Richer onboarding information, including product documentation and design artefacts, is the next lever for improving those cases.

03 Technique

Prompting Technique: Where the First Big Gains Came From

Before comparing models, we explored how much prompting technique alone could move the needle. Three approaches were tested on the same task, with the same model.

Zero-shot prompting (a basic instruction with no examples) achieved 63% accuracy. A meaningful improvement over the old system, but far from reliable.

Few-shot prompting (the same instruction with representative examples included) lifted that to 66%. Better, but the gain was smaller than expected. Examples help, but this task is too context-sensitive for examples alone to resolve the hard cases.

Advanced combined techniques (a structured prompt using meta-prompting, combined with chain-of-thought which was enabled by activating the models in-built thinking ability.) brought accuracy to 86%. This is the prompt architecture we then applied in the formal model comparison benchmark.

The implication is significant. An 86% result from an advanced prompt configuration versus 63% from a basic one, using the same model, is a larger delta than the 11-point gap between the best and worst models in the final benchmark. Prompt engineering is not a secondary concern for security-critical workflows; it is at least as important as model selection.

Extended thinking deserves specific mention. Enabling the model to reason through the problem step-by-step before committing to an answer produced the single largest single-variable improvement we observed. Zero-shot accuracy jumped from below 50% to approximately 60% with thinking mode enabled and nothing else changed. For a classification task that requires reasoning under genuine ambiguity, extended thinking is not a nice-to-have. It is foundational.

04 Results

Benchmarking Five Models

With the prompt architecture established, we ran a formal benchmark across five models. Each was given all tickets using the advanced combined prompt with extended thinking enabled at a budget of 8,000 tokens. Temperature was set to 1 across all runs, producing 180 outputs in total.

Claude Opus 4.6 leads clearly. All three Anthropic models occupy the top three positions. The gap between the best and worst performers is approximately 11 percentage points: meaningful when the decisions these models make determine whether a security test happens at all.

One result worth highlighting: despite being a newer release, Opus 4.7 scores 5.5 points lower than Opus 4.6. This is likely a result of the differences in how both models were trained. Opus 4.6 was designed with improvements to general reasoning and conversational tasks, essentially making it better at inferring information from prompts that might be more ambiguous in their nature.

With 4.7, the model was trained largely for agentic workflows where prompts are treated more literally, without attempting to infer what the user meant by those instructions.

It's this difference in intended use case that likely causes Opus 4.6 to perform better, despite being an older version. This highlights the importance in evaluating models against your specific problem, rather than relying on general leaderboard rankings in understanding which model is fit for purpose.

05 Analysis

Accuracy Alone Is Not the Whole Story

For a security triage task, the type of error matters as much as its rate.

A false positive flags a change as needing a security test when it does not. The cost is unnecessary work for testing teams and, over time, erosion of trust in the system. Teams start to override it. The signal degrades.

A false negative misses a change that does need a security test. This is the costlier failure mode in a risk context: a meaningful vulnerability slips through because the triage step decided it was not worth looking at.

Qwen3 Coder Next was observed to have a conservative bias. Its precision is high (0.85) because it rarely flags something unnecessarily, but its recall is low (0.65) because it misses real risk. In a security context, that is the wrong tradeoff. A model that quietly decides a risky change does not need testing is more dangerous than one that occasionally generates extra work.

Opus 4.7 has the opposite tendency: aggressive over-flagging, producing five false positives. More noise, more unnecessary work, but fewer coverage gaps.

Claude Opus 4.6 offers the best balance: three false positives and two false negatives. Claude Sonnet 4.6 produces a perfectly symmetric error profile, making it a credible alternative when operational cost is a constraint.

06 Implications

What Good Looks Like in Practice

Two findings from the benchmark are worth noting beyond the headline accuracy figures.

Every model in this evaluation produced structurally valid outputs across all 180 runs. Not a single response failed to parse. For production implementation, this matters: reliable structured output is not a given with LLMs, and achieving it consistently at temperature 1 demonstrates that the right prompt architecture can make these models behave predictably as components in an automated pipeline.

The not_enough_information verdict was used sparingly: only 3.3% of all responses, distributed fairly evenly across models. Models are generally willing to make a call on the available evidence rather than abstaining. That is the correct behaviour for a triage system. Excessive abstention shifts the decision burden back to humans without adding value, which is precisely the problem we are trying to solve.

07 Integration

How This Feeds into the Cytix Pipeline

This research sits at the first decision point in the Cytix risk review pipeline. The testability classification determines which changes progress to threat modelling, test plan generation, and execution. Accuracy here is not academic. A false negative at this stage means a security-relevant change never gets a test. A persistent pattern of false positives means testing teams face unnecessary work, and the pressure to override the system grows.

Extended thinking mode is being integrated into the live pipeline for steps requiring heavier reasoning, as well as knowledge retrieval capabilities to expand the context available to models. It also highlights a key outcome of the research: context = better results.

The 14% error rate that remains is not uniformly distributed across easy and hard cases. The tickets where models consistently disagree with the ground truth are, in most cases, edge cases that depend on context the ticket does not contain: architectural decisions, product-level assumptions, trust boundary choices that live outside the change itself. The next improvement vector is richer context at onboarding. Better product documentation, architecture notes, and design artefacts provided during platform setup will narrow that gap.

08 Commitment

On Transparency

We are publishing this research for a specific reason. Security vendors routinely make claims about AI accuracy without showing their working. We think that is a problem.

Every claim Cytix makes about the quality of its security judgement should be traceable back to evidence. This benchmark is one piece of that evidence. The methodology is documented. The dataset was built from real customer tickets. The ground truth was set by human experts. The results are presented without curation.

86.1% accuracy on a hard classification task, with a well-balanced error profile, from real tickets measured against human-labelled ground truth, is a result we are confident standing behind. It is also a baseline, not a ceiling. The work to improve it is already underway.

The full dataset is available on request.

09 About

About Cytix

The research above is not a theoretical exercise. The triage step it evaluates sits at the first decision point in a live production pipeline.

Cytix is how organisations qualify and act on software change risk. It starts with real work: a ticket, a pull request, a release, a deployment request, a remediation item, a configuration update, an infrastructure change. Those artefacts contain useful context, but that context is often fragmented, missed, or lost as work moves forward.

Cytix assembles the relevant context around the change: the affected asset, application knowledge, previous findings, policies, repository signal, and workflow metadata. It then qualifies whether the change matters, what security response is proportionate, who or what should act, and what evidence should remain attached to the decision.

The right response is not always a test. It might be more context, a human review, a threat model, a code review, a specific validation step, penetration testing, remediation confirmation, or an accepted-risk record. Testing is part of the story. Cytix decides when testing is the right response to a change, rather than running tests because a schedule says so.

The result: security, engineering, product, risk, and audit teams work from the same view of what changed, why it matters, what should happen next, and what evidence remains.

Security understood.