The Detection benchmark scores three lanes on 1,205 evaluation units:
All LLMs run in the same detection harness: four analyst personas with network-flow tools submitting per-unit verdicts. LogLM scores sequences directly, projected onto the eval-unit class prior.
soc_analystsoc_analystthreat_analystthreat_analystsoc_analystsoc_analystOverall across all 1,205 units. LLM bars are the mean across 4 personas; LogLM is the single classifier. The per-flow / verdict F1 chart further down responds to the split tabs.
* glm-5.2 reached only 27% of the 1,205 shared units (328 of 1,205); bars use that subset and may not be representative of the model at full coverage.
benign: every flow in the evaluation unit was benign. In the chart and table below, Verdict acc is the true-negative rate (specificity).
SOCBench starts with a labeled NetFlow corpus that is provided here. Similar approaches on other logs are planned. From these logs a per-provider score is calculated in seven deterministic steps, enabling easy comparisons. Four analyst personas with different tool budgets are defined and work the same evaluation units; submissions are scored against held-out gold labels.
Each persona has its own turn / tool-call / wall-clock budget and a defined toolset. Higher-tier personas inherit everything below them. The SOC analyst is the lean baseline; the detection engineer carries the full toolkit.
| Tool | soc_analyst | threat_analyst | adversary_hunter | detection_engineer |
|---|---|---|---|---|
list_pairs | ● | ● | ● | ● |
get_pair_timeline | ● | ● | ● | ● |
get_flows | ● | ● | ● | ● |
host_rollup | ● | ● | ● | ● |
submit_assessment | ● | ● | ● | ● |
top_destinations | · | ● | ● | ● |
pair_stats | · | ● | ● | ● |
port_proto_matrix | · | · | ● | ● |
rarity_stats | · | · | ● | ● |
| Budget (turns / calls / wall) | 4 / 6 / 60s | 8 / 12 / 120s | 10 / 16 / 150s | 12 / 20 / 180s |
Models in this run:
anthropic = claude-opus-4-7
gemini = gemini-2.5-pro
openai = gpt-5.4.
OpenAI receives a 1.5× persona-budget multiplier (its reasoning chains are chattier; without it, large host_egress units exhaust budgets before submission). Disclosed for fair comparison.
Blogs and docs explaining the methodology in more depth, including the corpus breakdown, eval-unit construction, scoring-lens math, run-metadata schema, and deterministic-replay tuple, are coming soon. If you look at the underlying GitHub you can see some of this explained in /docs and in the readme. This simple website is by design higher level.
All 25 configurations plotted on the F1 / cost frontier: 6 LLMs × 4 personas, plus LogLM. Winners on each contract dimension called out below (LLMs only, since LogLM has no personas).
At one million alerts a day, the cheapest LLM stack ($0.057 per alert) is roughly $60,000 of inference every day, before a single analyst looks at the output. At telco scale (a large telco active in the Vigil community reports approximately 70 million flow events per minute, which batches into roughly 2 billion alerts per day at 50 flows per alert), the LLM bill runs to hundreds of millions of dollars per day ($114M at the cheapest provider, ~$302M at the priciest).
For each contract dimension, the winning (LLM × persona) configuration across all 24 combinations. LogLM excluded here since it has no personas.
soc_analystsoc_analystdetection_engineerEvery number above traces back to one of these three runs. Full artifacts and prompts are in the repo.
| Provider | Run ID | Units | Renderings | Cost |
|---|---|---|---|---|
| anthropic | 20260604T011942Z_full_main_anthropic_4bc5181b_632cbb | 1,205 | 4,814 | $724.82 |
| openai | 20260604T012213Z_full_main_openai_4bc5181b_0b1f4c | 1,500 | 6,000 | $328.59 |
| gemini | 20260604T012407Z_full_main_gemini_4bc5181b_e4a9c6 | 1,500 | 6,000 | $355.13 |
A full run (~1,500 units × 4 personas × 3 providers) costs roughly $1,300 to $1,500 USD in LLM spend at the pricing snapshot dates in this report. A smoke run (8 units) is under $10.